Yeah one of the worries here is that while this makes a lot of sense for parquet, it doesn’t make any sense for Avro. We were mostly targeting a more conservative set of changes and then more global modifications later.
On Wed, Aug 5, 2026 at 9:41 AM Daniel Weeks <[email protected]> wrote: > Sorry about the confusion, I think my phrasing wasn't quite accurate, but > the point remains. > > The referenced PR for integrating this directly in the ManifestFiles > class, which narrowly targets the metadata path, not making this a general > FileIO capability, but rather a bespoke shortcircut in the metadata path. > > Prefetching like this would potentially benefit any file operation > (particularly with small parquet files), but that's not what's proposed. > > I'm suggesting integrating this into the existing FileIO implementations > so any InputFile::newStream would be accelerated. The difference is making > it native to the FileIO as opposed to wrapping the stream after. > > Hopefully that clarifies, > -Dan > > On Wed, Aug 5, 2026 at 5:13 AM Russell Spitzer <[email protected]> > wrote: > >> Could you elaborate a bit more Dan? This isn’t part of the fileio, it’s a >> separate file class (much like encrypting file) which is used by all IO >> like the encryption implementation. >> >> On Tue, Aug 4, 2026 at 1:34 PM Daniel Weeks <[email protected]> wrote: >> >>> Hey Varun, >>> >>> I noticed a couple weeks ago when this went in and was a little confused >>> at the time because it wasn't wired in and didn't appear to be generically >>> useable, so I'm excited to see this follow up. >>> >>> I'm a little concerned based on my impression from the PRs that we're >>> thinking about integrating this in the wrong way. Rather than inserting a >>> specific FileIO implementation in sections of the code path, it seems this >>> would more generically apply as a base implementation for any FileIO. >>> >>> I think my preference would be that if you enable eager loading, it >>> would apply to any existing FileIO implementation and any access. Then >>> it's just a matter of tuning the threshold where the implementation >>> switches from eager fetching to standard/vectored IO paths. >>> >>> -Dan >>> >>> On Tue, Aug 4, 2026 at 11:27 AM vaquar khan <[email protected]> >>> wrote: >>> >>>> This is awesome and really valuable, request others to review and >>>> conclude ,if needs plz perform own benchmark. >>>> >>>> Regards, >>>> Viquar Khan >>>> >>>> On Mon, Aug 3, 2026, 11:15 AM Kevin Liu <[email protected]> wrote: >>>> >>>>> Thanks for the great work, this is super exciting. I've been looking >>>>> at object storage optimizations lately, I'll find some time this week to >>>>> take a look. >>>>> >>>>> On Mon, Aug 3, 2026 at 8:19 AM Varun Lakhyani < >>>>> [email protected]> wrote: >>>>> >>>>>> I am looking for views on keeping the one flag true by default to use >>>>>> EagerInputFile in ManifestFiles read path (Major benefits in v4 Parquet >>>>>> Manifests). >>>>>> We have ready to run benchmark [1] and two independent sets of >>>>>> results showing similar benefits ~25 - 55% reduction in reading Parquet >>>>>> manifest depending on machine and file. >>>>>> >>>>>> I would appreciate the community's look and feedback on this - PR >>>>>> having changes [2] and whether we can enable this path by default. >>>>>> >>>>>> [1] https://github.com/varun-lakhyani/iceberg/pull/1 >>>>>> [2] https://github.com/apache/iceberg/pull/17284 >>>>>> >>>>>> On Wed, Jul 29, 2026 at 6:29 AM Russell Spitzer < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I'm not sure if anyone else has checked this out yet, but I think >>>>>>> it's a really exciting improvement. We should seriously consider making >>>>>>> this a default feature in the next release. Or potentially just have it >>>>>>> always be on. >>>>>>> >>>>>>> On Tue, Jul 28, 2026 at 4:58 PM Varun Lakhyani < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I think we can flag property to enable this as default true, It >>>>>>>> would be a great improvement for v4 parquet manifest. >>>>>>>> Tried to do benchmarkings as extensive as possible. >>>>>>>> I would love to hear thoughts on this. >>>>>>>> >>>>>>>> Please review the PR once and would appreciate feedback. >>>>>>>> >>>>>>>> On Mon, Jul 20, 2026 at 10:02 PM Varun Lakhyani < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hello Everyone, >>>>>>>>> >>>>>>>>> I evaluated using EagerInputFile in v4 manifest reader path >>>>>>>>> (parquet) using ManifestBenchmark and S3 as object store and EC2 >>>>>>>>> machine >>>>>>>>> for jmh benchmarking. >>>>>>>>> >>>>>>>>> This is to make sure we are doing a proper checklist while wiring >>>>>>>>> this EagerInputFile path here, >>>>>>>>> Detailed benchmarking setup and results are commented on in PR[1]. >>>>>>>>> Using an EC2 machine in the same region (ap-south-1) gives >>>>>>>>> *~28.4%* to *~42.7%* betterment while comparing against baseline >>>>>>>>> default while benchmarking. >>>>>>>>> >>>>>>>>> Please provide review and feedback. >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> [1] https://github.com/apache/iceberg/pull/17284 >>>>>>>>> -- >>>>>>>>> Lakhyani Varun >>>>>>>>> Indian Institute of Technology Roorkee >>>>>>>>> Contact: +91 96246 46174 >>>>>>>>> >>>>>>>>>
