Benchmark results for avro using ManifestBenchmarks (I already ran this
along with parquet last time) AVRO v4 (Non-Partitioned)
Number of Columns
Default (s/op)
Eager Fetch (s/op)
Latency Reduction
10
0.086
0.095
-10.5%
50
0.086
0.082
4.7%
100
0.080
0.076
5.0%
AVRO v4 (Partitioned)
Number of Columns
Default (s/op)
Eager Fetch (s/op)
Latency Reduction
10
0.092
0.095
-3.3%
50
0.086
0.086
0.0%
100
0.094
0.088
6.4%
After ignoring jmh noise, The results appear largely neutral overall. Just
that this eager path temporarily retains the entire file in memory, which
is then reclaimed through normal garbage collection.
Just to confirm we are talking about wiring inside like
S3Inputfile.newStream and similar at ADLSInputfile / GCSInputFile just as I
have shown below. (what is exactly in eagerInputFile wrapper)
// S3InputFile.newStream()
if (eager && getLength() <= threshold) { // + optional format-aware
check if needed
byte[] bytes = readFully // single GET
return new EagerInputStream(bytes);
}
return new S3InputStream(...); // normal path
this will skip wrapping and just returns eagerInputStream instead of like
S3InputStream/ADLSInputStream.
The crux of this implementation will be we have to put it inside all
InputFile::newStream at which we want this eager to be enabled and this
will be format independent unless we put the check specifically.
Happy to proceed with either path everyone agrees on - global change at
FileIO implementation layer or small wiring for now (Manifest and spark
readers are 2 points I can think of will be most beneficial).
I would appreciate everyone's thoughts on which direction makes the most
sense.
On Thu, Aug 6, 2026 at 12:30 AM Daniel Weeks <[email protected]> wrote:
> I'm not convinced it's actually worse for Avro. With Avro, you typically
> read the entire file if you intend to open it at all. There's no real
> skipping or ranged projection happening, so I'm not convinced that a full
> file fetch would be net slower than the typical read path, which may
> require multiple incremental fetches (it might even help some of the byte
> skipping paths).
>
> That would be good to benchmark, but I know multiple implementations have
> this exact type of optimization. If we think we need file type specific
> behavior, we could incorporate that into the FileIO abstraction as well.
>
> On Wed, Aug 5, 2026 at 8:02 AM Russell Spitzer <[email protected]>
> wrote:
>
>> Yeah one of the worries here is that while this makes a lot of sense for
>> parquet, it doesn’t make any sense for Avro. We were mostly targeting a
>> more conservative set of changes and then more global modifications later.
>>
>> On Wed, Aug 5, 2026 at 9:41 AM Daniel Weeks <[email protected]> wrote:
>>
>>> Sorry about the confusion, I think my phrasing wasn't quite accurate,
>>> but the point remains.
>>>
>>> The referenced PR for integrating this directly in the ManifestFiles
>>> class, which narrowly targets the metadata path, not making this a general
>>> FileIO capability, but rather a bespoke shortcircut in the metadata path.
>>>
>>> Prefetching like this would potentially benefit any file operation
>>> (particularly with small parquet files), but that's not what's proposed.
>>>
>>> I'm suggesting integrating this into the existing FileIO implementations
>>> so any InputFile::newStream would be accelerated. The difference is making
>>> it native to the FileIO as opposed to wrapping the stream after.
>>>
>>> Hopefully that clarifies,
>>> -Dan
>>>
>>> On Wed, Aug 5, 2026 at 5:13 AM Russell Spitzer <
>>> [email protected]> wrote:
>>>
>>>> Could you elaborate a bit more Dan? This isn’t part of the fileio, it’s
>>>> a separate file class (much like encrypting file) which is used by all IO
>>>> like the encryption implementation.
>>>>
>>>> On Tue, Aug 4, 2026 at 1:34 PM Daniel Weeks <[email protected]> wrote:
>>>>
>>>>> Hey Varun,
>>>>>
>>>>> I noticed a couple weeks ago when this went in and was a little
>>>>> confused at the time because it wasn't wired in and didn't appear to be
>>>>> generically useable, so I'm excited to see this follow up.
>>>>>
>>>>> I'm a little concerned based on my impression from the PRs that we're
>>>>> thinking about integrating this in the wrong way. Rather than inserting a
>>>>> specific FileIO implementation in sections of the code path, it seems this
>>>>> would more generically apply as a base implementation for any FileIO.
>>>>>
>>>>> I think my preference would be that if you enable eager loading, it
>>>>> would apply to any existing FileIO implementation and any access. Then
>>>>> it's just a matter of tuning the threshold where the implementation
>>>>> switches from eager fetching to standard/vectored IO paths.
>>>>>
>>>>> -Dan
>>>>>
>>>>> On Tue, Aug 4, 2026 at 11:27 AM vaquar khan <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> This is awesome and really valuable, request others to review and
>>>>>> conclude ,if needs plz perform own benchmark.
>>>>>>
>>>>>> Regards,
>>>>>> Viquar Khan
>>>>>>
>>>>>> On Mon, Aug 3, 2026, 11:15 AM Kevin Liu <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the great work, this is super exciting. I've been looking
>>>>>>> at object storage optimizations lately, I'll find some time this week to
>>>>>>> take a look.
>>>>>>>
>>>>>>> On Mon, Aug 3, 2026 at 8:19 AM Varun Lakhyani <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> I am looking for views on keeping the one flag true by default to
>>>>>>>> use EagerInputFile in ManifestFiles read path (Major benefits in v4
>>>>>>>> Parquet
>>>>>>>> Manifests).
>>>>>>>> We have ready to run benchmark [1] and two independent sets of
>>>>>>>> results showing similar benefits ~25 - 55% reduction in reading Parquet
>>>>>>>> manifest depending on machine and file.
>>>>>>>>
>>>>>>>> I would appreciate the community's look and feedback on this - PR
>>>>>>>> having changes [2] and whether we can enable this path by default.
>>>>>>>>
>>>>>>>> [1] https://github.com/varun-lakhyani/iceberg/pull/1
>>>>>>>> [2] https://github.com/apache/iceberg/pull/17284
>>>>>>>>
>>>>>>>> On Wed, Jul 29, 2026 at 6:29 AM Russell Spitzer <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> I'm not sure if anyone else has checked this out yet, but I think
>>>>>>>>> it's a really exciting improvement. We should seriously consider
>>>>>>>>> making
>>>>>>>>> this a default feature in the next release. Or potentially just have
>>>>>>>>> it
>>>>>>>>> always be on.
>>>>>>>>>
>>>>>>>>> On Tue, Jul 28, 2026 at 4:58 PM Varun Lakhyani <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> I think we can flag property to enable this as default true, It
>>>>>>>>>> would be a great improvement for v4 parquet manifest.
>>>>>>>>>> Tried to do benchmarkings as extensive as possible.
>>>>>>>>>> I would love to hear thoughts on this.
>>>>>>>>>>
>>>>>>>>>> Please review the PR once and would appreciate feedback.
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 20, 2026 at 10:02 PM Varun Lakhyani <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Everyone,
>>>>>>>>>>>
>>>>>>>>>>> I evaluated using EagerInputFile in v4 manifest reader path
>>>>>>>>>>> (parquet) using ManifestBenchmark and S3 as object store and EC2
>>>>>>>>>>> machine
>>>>>>>>>>> for jmh benchmarking.
>>>>>>>>>>>
>>>>>>>>>>> This is to make sure we are doing a proper checklist while
>>>>>>>>>>> wiring this EagerInputFile path here,
>>>>>>>>>>> Detailed benchmarking setup and results are commented on in
>>>>>>>>>>> PR[1].
>>>>>>>>>>> Using an EC2 machine in the same region (ap-south-1) gives
>>>>>>>>>>> *~28.4%* to *~42.7%* betterment while comparing against
>>>>>>>>>>> baseline default while benchmarking.
>>>>>>>>>>>
>>>>>>>>>>> Please provide review and feedback.
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> [1] https://github.com/apache/iceberg/pull/17284
>>>>>>>>>>> --
>>>>>>>>>>> Lakhyani Varun
>>>>>>>>>>> Indian Institute of Technology Roorkee
>>>>>>>>>>> Contact: +91 96246 46174
>>>>>>>>>>>
>>>>>>>>>>>
