Could you elaborate a bit more Dan? This isn’t part of the fileio, it’s a separate file class (much like encrypting file) which is used by all IO like the encryption implementation.
On Tue, Aug 4, 2026 at 1:34 PM Daniel Weeks <[email protected]> wrote: > Hey Varun, > > I noticed a couple weeks ago when this went in and was a little confused > at the time because it wasn't wired in and didn't appear to be generically > useable, so I'm excited to see this follow up. > > I'm a little concerned based on my impression from the PRs that we're > thinking about integrating this in the wrong way. Rather than inserting a > specific FileIO implementation in sections of the code path, it seems this > would more generically apply as a base implementation for any FileIO. > > I think my preference would be that if you enable eager loading, it would > apply to any existing FileIO implementation and any access. Then it's just > a matter of tuning the threshold where the implementation switches from > eager fetching to standard/vectored IO paths. > > -Dan > > On Tue, Aug 4, 2026 at 11:27 AM vaquar khan <[email protected]> wrote: > >> This is awesome and really valuable, request others to review and >> conclude ,if needs plz perform own benchmark. >> >> Regards, >> Viquar Khan >> >> On Mon, Aug 3, 2026, 11:15 AM Kevin Liu <[email protected]> wrote: >> >>> Thanks for the great work, this is super exciting. I've been looking at >>> object storage optimizations lately, I'll find some time this week to take >>> a look. >>> >>> On Mon, Aug 3, 2026 at 8:19 AM Varun Lakhyani <[email protected]> >>> wrote: >>> >>>> I am looking for views on keeping the one flag true by default to use >>>> EagerInputFile in ManifestFiles read path (Major benefits in v4 Parquet >>>> Manifests). >>>> We have ready to run benchmark [1] and two independent sets of results >>>> showing similar benefits ~25 - 55% reduction in reading Parquet manifest >>>> depending on machine and file. >>>> >>>> I would appreciate the community's look and feedback on this - PR >>>> having changes [2] and whether we can enable this path by default. >>>> >>>> [1] https://github.com/varun-lakhyani/iceberg/pull/1 >>>> [2] https://github.com/apache/iceberg/pull/17284 >>>> >>>> On Wed, Jul 29, 2026 at 6:29 AM Russell Spitzer < >>>> [email protected]> wrote: >>>> >>>>> I'm not sure if anyone else has checked this out yet, but I think it's >>>>> a really exciting improvement. We should seriously consider making this a >>>>> default feature in the next release. Or potentially just have it always be >>>>> on. >>>>> >>>>> On Tue, Jul 28, 2026 at 4:58 PM Varun Lakhyani < >>>>> [email protected]> wrote: >>>>> >>>>>> I think we can flag property to enable this as default true, It would >>>>>> be a great improvement for v4 parquet manifest. >>>>>> Tried to do benchmarkings as extensive as possible. >>>>>> I would love to hear thoughts on this. >>>>>> >>>>>> Please review the PR once and would appreciate feedback. >>>>>> >>>>>> On Mon, Jul 20, 2026 at 10:02 PM Varun Lakhyani < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hello Everyone, >>>>>>> >>>>>>> I evaluated using EagerInputFile in v4 manifest reader path >>>>>>> (parquet) using ManifestBenchmark and S3 as object store and EC2 machine >>>>>>> for jmh benchmarking. >>>>>>> >>>>>>> This is to make sure we are doing a proper checklist while wiring >>>>>>> this EagerInputFile path here, >>>>>>> Detailed benchmarking setup and results are commented on in PR[1]. >>>>>>> Using an EC2 machine in the same region (ap-south-1) gives *~28.4%* >>>>>>> to *~42.7%* betterment while comparing against baseline default >>>>>>> while benchmarking. >>>>>>> >>>>>>> Please provide review and feedback. >>>>>>> Thanks >>>>>>> >>>>>>> [1] https://github.com/apache/iceberg/pull/17284 >>>>>>> -- >>>>>>> Lakhyani Varun >>>>>>> Indian Institute of Technology Roorkee >>>>>>> Contact: +91 96246 46174 >>>>>>> >>>>>>>
