Yes this change is split across 2 PRs - implementation in earlier PR, and now the wiring part.
Agreed on making this generic - I am working on an EagerFileIO that covers all access paths rather than just manifests and keeps this logic away from code path. Will share the further updates soon. On Wed, Aug 5, 2026 at 12:05 AM Daniel Weeks <[email protected]> wrote: > Hey Varun, > > I noticed a couple weeks ago when this went in and was a little confused > at the time because it wasn't wired in and didn't appear to be generically > useable, so I'm excited to see this follow up. > > I'm a little concerned based on my impression from the PRs that we're > thinking about integrating this in the wrong way. Rather than inserting a > specific FileIO implementation in sections of the code path, it seems this > would more generically apply as a base implementation for any FileIO. > > I think my preference would be that if you enable eager loading, it would > apply to any existing FileIO implementation and any access. Then it's just > a matter of tuning the threshold where the implementation switches from > eager fetching to standard/vectored IO paths. > > -Dan > > On Tue, Aug 4, 2026 at 11:27 AM vaquar khan <[email protected]> wrote: > >> This is awesome and really valuable, request others to review and >> conclude ,if needs plz perform own benchmark. >> >> Regards, >> Viquar Khan >> >> On Mon, Aug 3, 2026, 11:15 AM Kevin Liu <[email protected]> wrote: >> >>> Thanks for the great work, this is super exciting. I've been looking at >>> object storage optimizations lately, I'll find some time this week to take >>> a look. >>> >>> On Mon, Aug 3, 2026 at 8:19 AM Varun Lakhyani <[email protected]> >>> wrote: >>> >>>> I am looking for views on keeping the one flag true by default to use >>>> EagerInputFile in ManifestFiles read path (Major benefits in v4 Parquet >>>> Manifests). >>>> We have ready to run benchmark [1] and two independent sets of results >>>> showing similar benefits ~25 - 55% reduction in reading Parquet manifest >>>> depending on machine and file. >>>> >>>> I would appreciate the community's look and feedback on this - PR >>>> having changes [2] and whether we can enable this path by default. >>>> >>>> [1] https://github.com/varun-lakhyani/iceberg/pull/1 >>>> [2] https://github.com/apache/iceberg/pull/17284 >>>> >>>> On Wed, Jul 29, 2026 at 6:29 AM Russell Spitzer < >>>> [email protected]> wrote: >>>> >>>>> I'm not sure if anyone else has checked this out yet, but I think it's >>>>> a really exciting improvement. We should seriously consider making this a >>>>> default feature in the next release. Or potentially just have it always be >>>>> on. >>>>> >>>>> On Tue, Jul 28, 2026 at 4:58 PM Varun Lakhyani < >>>>> [email protected]> wrote: >>>>> >>>>>> I think we can flag property to enable this as default true, It would >>>>>> be a great improvement for v4 parquet manifest. >>>>>> Tried to do benchmarkings as extensive as possible. >>>>>> I would love to hear thoughts on this. >>>>>> >>>>>> Please review the PR once and would appreciate feedback. >>>>>> >>>>>> On Mon, Jul 20, 2026 at 10:02 PM Varun Lakhyani < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hello Everyone, >>>>>>> >>>>>>> I evaluated using EagerInputFile in v4 manifest reader path >>>>>>> (parquet) using ManifestBenchmark and S3 as object store and EC2 machine >>>>>>> for jmh benchmarking. >>>>>>> >>>>>>> This is to make sure we are doing a proper checklist while wiring >>>>>>> this EagerInputFile path here, >>>>>>> Detailed benchmarking setup and results are commented on in PR[1]. >>>>>>> Using an EC2 machine in the same region (ap-south-1) gives *~28.4%* >>>>>>> to *~42.7%* betterment while comparing against baseline default >>>>>>> while benchmarking. >>>>>>> >>>>>>> Please provide review and feedback. >>>>>>> Thanks >>>>>>> >>>>>>> [1] https://github.com/apache/iceberg/pull/17284 >>>>>>> -- >>>>>>> Lakhyani Varun >>>>>>> Indian Institute of Technology Roorkee >>>>>>> Contact: +91 96246 46174 >>>>>>> >>>>>>>
