A few months ago in a PR comment [1] Steve Loughran mentioned that tracking
the known length of a file could be helpful. I was intrigued by that idea
and explored it this week. I riffed on the idea with Codex and ended up
with a possible solution. I'm bringing it up here because I'm not yet
confident enough in the change for a PR, but I can certainly open one.
Here's my understanding that I'm trying to validate:
The basic idea is that starting with Hadoop 3.3.5 (for reference we
currently depend on 3.3.0) a hint called `fs.option.openfile.length` can be
supplied when building a SeekableInputStream instead of supplying a full
FileStat object. Catalogs like Iceberg (and I assume others) currently
construct the stat object, which entails a cloud store metadata lookup, but
they already track the length and could use that instead and save the
lookup cost.
So what if we add this method to HadoopInputFile?
HadoopInputFile fromPath(Path path, long length, Configuration conf)
and then the newStream() method uses the stat or the length depending on
which `from` method was used.
Stylistically, with this change instead of:
HadoopInputFile.fromStatus(stat, conf)`
Iceberg could
HadoopInputFile.fromPath(
input.getPath(),
input.getLength(),
input.getConf())
My understanding: If the hadoop version used is at least 3.3.0 but less
than 3.3.5 then the option is ignored and a metadata lookup made anyway.
Below 3.3.0 this could be an error, but we're advertising a minimum
requirement of 3.3.0 so that seems acceptable and not a breaking change.
My PR would also include bumping the dependency to 3.3.5
Does this seem like a good idea? Am I missing anything? Is this idea
solid enough to open a PR and proceed?
Thanks! -- Aaron
[1] https://github.com/apache/parquet-java/pull/3395#issuecomment-4268040711
--
Aaron Niskode-Dossett, Data Engineering -- Etsy
