I'd really like to understand the pros of putting this in FileIO a bit better.
If we want to achieve global coverage immediately, we could just jump to Parquet.java <https://github.com/apache/iceberg/blob/3d682a3b65/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L1269-L1271> instead of doing a manifest specific code change. That would cover all usages we care about and avoid adding complexity to the IO implementation for what is essentially a performance fix for parquet-java. Ideally, this should be a fix in parquet-java directly, so keeping it contained to our Parquet reader code feels like the right place for me. There's no reason for us to trigger the same path for, say, a Puffin file or metadata.json. On Wed, Aug 5, 2026 at 7:30 PM Daniel Weeks <[email protected]> wrote: > We almost always have the file length from metadata (we already plumbed > this through to avoid additional head requests). > > We can infer the type from the path, but it doesn't seem like it would be > necessary if Avro is a wash and parquet is faster. > > > > On Wed, Aug 5, 2026, 2:46 PM Russell Spitzer <[email protected]> > wrote: > >> It just seems a lot more complicated to me that the decision on whether >> to prefetch and catch the file would be made by the fileio and not the tool >> opening the file. I wasn’t saying it would be slower, just unnecessary. If >> you check the pr, Varun actually did this benchmark already and Avro is >> basically unchanged. >> >> Are you saying fileio should know the size of the file and its type when >> opening? At the moment it really is pretty opaque about that sort of thing. >> >> On Wed, Aug 5, 2026 at 1:43 PM Daniel Weeks <[email protected]> wrote: >> >>> I'm not convinced it's actually worse for Avro. With Avro, you >>> typically read the entire file if you intend to open it at all. There's no >>> real skipping or ranged projection happening, so I'm not convinced that a >>> full file fetch would be net slower than the typical read path, which may >>> require multiple incremental fetches (it might even help some of the byte >>> skipping paths). >>> >>> That would be good to benchmark, but I know multiple implementations >>> have this exact type of optimization. If we think we need file type >>> specific behavior, we could incorporate that into the FileIO abstraction as >>> well. >>> >>> On Wed, Aug 5, 2026 at 8:02 AM Russell Spitzer < >>> [email protected]> wrote: >>> >>>> Yeah one of the worries here is that while this makes a lot of sense >>>> for parquet, it doesn’t make any sense for Avro. We were mostly targeting a >>>> more conservative set of changes and then more global modifications later. >>>> >>>> On Wed, Aug 5, 2026 at 9:41 AM Daniel Weeks <[email protected]> wrote: >>>> >>>>> Sorry about the confusion, I think my phrasing wasn't quite accurate, >>>>> but the point remains. >>>>> >>>>> The referenced PR for integrating this directly in the ManifestFiles >>>>> class, which narrowly targets the metadata path, not making this a general >>>>> FileIO capability, but rather a bespoke shortcircut in the metadata path. >>>>> >>>>> Prefetching like this would potentially benefit any file operation >>>>> (particularly with small parquet files), but that's not what's proposed. >>>>> >>>>> I'm suggesting integrating this into the existing FileIO >>>>> implementations so any InputFile::newStream would be accelerated. The >>>>> difference is making it native to the FileIO as opposed to wrapping the >>>>> stream after. >>>>> >>>>> Hopefully that clarifies, >>>>> -Dan >>>>> >>>>> On Wed, Aug 5, 2026 at 5:13 AM Russell Spitzer < >>>>> [email protected]> wrote: >>>>> >>>>>> Could you elaborate a bit more Dan? This isn’t part of the fileio, >>>>>> it’s a separate file class (much like encrypting file) which is used by >>>>>> all >>>>>> IO like the encryption implementation. >>>>>> >>>>>> On Tue, Aug 4, 2026 at 1:34 PM Daniel Weeks <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hey Varun, >>>>>>> >>>>>>> I noticed a couple weeks ago when this went in and was a little >>>>>>> confused at the time because it wasn't wired in and didn't appear to be >>>>>>> generically useable, so I'm excited to see this follow up. >>>>>>> >>>>>>> I'm a little concerned based on my impression from the PRs that >>>>>>> we're thinking about integrating this in the wrong way. Rather than >>>>>>> inserting a specific FileIO implementation in sections of the code >>>>>>> path, it >>>>>>> seems this would more generically apply as a base implementation for any >>>>>>> FileIO. >>>>>>> >>>>>>> I think my preference would be that if you enable eager loading, it >>>>>>> would apply to any existing FileIO implementation and any access. Then >>>>>>> it's just a matter of tuning the threshold where the implementation >>>>>>> switches from eager fetching to standard/vectored IO paths. >>>>>>> >>>>>>> -Dan >>>>>>> >>>>>>> On Tue, Aug 4, 2026 at 11:27 AM vaquar khan <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> This is awesome and really valuable, request others to review and >>>>>>>> conclude ,if needs plz perform own benchmark. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Viquar Khan >>>>>>>> >>>>>>>> On Mon, Aug 3, 2026, 11:15 AM Kevin Liu <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks for the great work, this is super exciting. I've been >>>>>>>>> looking at object storage optimizations lately, I'll find some time >>>>>>>>> this >>>>>>>>> week to take a look. >>>>>>>>> >>>>>>>>> On Mon, Aug 3, 2026 at 8:19 AM Varun Lakhyani < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> I am looking for views on keeping the one flag true by default to >>>>>>>>>> use EagerInputFile in ManifestFiles read path (Major benefits in v4 >>>>>>>>>> Parquet >>>>>>>>>> Manifests). >>>>>>>>>> We have ready to run benchmark [1] and two independent sets of >>>>>>>>>> results showing similar benefits ~25 - 55% reduction in reading >>>>>>>>>> Parquet >>>>>>>>>> manifest depending on machine and file. >>>>>>>>>> >>>>>>>>>> I would appreciate the community's look and feedback on this - PR >>>>>>>>>> having changes [2] and whether we can enable this path by default. >>>>>>>>>> >>>>>>>>>> [1] https://github.com/varun-lakhyani/iceberg/pull/1 >>>>>>>>>> [2] https://github.com/apache/iceberg/pull/17284 >>>>>>>>>> >>>>>>>>>> On Wed, Jul 29, 2026 at 6:29 AM Russell Spitzer < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I'm not sure if anyone else has checked this out yet, but I >>>>>>>>>>> think it's a really exciting improvement. We should seriously >>>>>>>>>>> consider >>>>>>>>>>> making this a default feature in the next release. Or potentially >>>>>>>>>>> just have >>>>>>>>>>> it always be on. >>>>>>>>>>> >>>>>>>>>>> On Tue, Jul 28, 2026 at 4:58 PM Varun Lakhyani < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> I think we can flag property to enable this as default true, It >>>>>>>>>>>> would be a great improvement for v4 parquet manifest. >>>>>>>>>>>> Tried to do benchmarkings as extensive as possible. >>>>>>>>>>>> I would love to hear thoughts on this. >>>>>>>>>>>> >>>>>>>>>>>> Please review the PR once and would appreciate feedback. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jul 20, 2026 at 10:02 PM Varun Lakhyani < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello Everyone, >>>>>>>>>>>>> >>>>>>>>>>>>> I evaluated using EagerInputFile in v4 manifest reader path >>>>>>>>>>>>> (parquet) using ManifestBenchmark and S3 as object store and EC2 >>>>>>>>>>>>> machine >>>>>>>>>>>>> for jmh benchmarking. >>>>>>>>>>>>> >>>>>>>>>>>>> This is to make sure we are doing a proper checklist while >>>>>>>>>>>>> wiring this EagerInputFile path here, >>>>>>>>>>>>> Detailed benchmarking setup and results are commented on in >>>>>>>>>>>>> PR[1]. >>>>>>>>>>>>> Using an EC2 machine in the same region (ap-south-1) gives >>>>>>>>>>>>> *~28.4%* to *~42.7%* betterment while comparing against >>>>>>>>>>>>> baseline default while benchmarking. >>>>>>>>>>>>> >>>>>>>>>>>>> Please provide review and feedback. >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> [1] https://github.com/apache/iceberg/pull/17284 >>>>>>>>>>>>> -- >>>>>>>>>>>>> Lakhyani Varun >>>>>>>>>>>>> Indian Institute of Technology Roorkee >>>>>>>>>>>>> Contact: +91 96246 46174 >>>>>>>>>>>>> >>>>>>>>>>>>>
