On second thought, I decided to open a PR. https://github.com/apache/parquet-java/pull/3678
On Sat, Jul 18, 2026 at 10:02 AM Aaron Niskode-Dossett < [email protected]> wrote: > A few months ago in a PR comment [1] Steve Loughran mentioned that > tracking the known length of a file could be helpful. I was intrigued by > that idea and explored it this week. I riffed on the idea with Codex and > ended up with a possible solution. I'm bringing it up here because I'm not > yet confident enough in the change for a PR, but I can certainly open one. > > Here's my understanding that I'm trying to validate: > > The basic idea is that starting with Hadoop 3.3.5 (for reference we > currently depend on 3.3.0) a hint called `fs.option.openfile.length` can be > supplied when building a SeekableInputStream instead of supplying a full > FileStat object. Catalogs like Iceberg (and I assume others) currently > construct the stat object, which entails a cloud store metadata lookup, but > they already track the length and could use that instead and save the > lookup cost. > > So what if we add this method to HadoopInputFile? > > HadoopInputFile fromPath(Path path, long length, Configuration conf) > > and then the newStream() method uses the stat or the length depending on > which `from` method was used. > > Stylistically, with this change instead of: > > HadoopInputFile.fromStatus(stat, conf)` > > Iceberg could > > HadoopInputFile.fromPath( > input.getPath(), > input.getLength(), > input.getConf()) > > My understanding: If the hadoop version used is at least 3.3.0 but less > than 3.3.5 then the option is ignored and a metadata lookup made anyway. > Below 3.3.0 this could be an error, but we're advertising a minimum > requirement of 3.3.0 so that seems acceptable and not a breaking change. > My PR would also include bumping the dependency to 3.3.5 > > Does this seem like a good idea? Am I missing anything? Is this idea > solid enough to open a PR and proceed? > > Thanks! -- Aaron > > [1] > https://github.com/apache/parquet-java/pull/3395#issuecomment-4268040711 > > -- > Aaron Niskode-Dossett, Data Engineering -- Etsy > -- Aaron Niskode-Dossett, Data Engineering -- Etsy
