Before doing any work there I would talk to the aws people, especially Danny C Jones, benchmark with the AWS input stream underneath (s3.analytics-accelerator.enabled = true), and someone should merge the PR there to wire its support for parquet vector reads through iceberg, for the 30% TPC-DS benchmark improvements the vector reads deliver against S3.
the accelerator stream will already be prefetching ahead of that 8 byte footer (I think its about 64k), so that average footers are retrieved in that single GET. and then if if you don't do vector IO it looks at your read pattern, compares with the parquet schema and predicts your next read. Vector is better as there's no need to predict and you can do scatter/gather read straight into separate buffers. Once you've got that wired up and benchmarked, then you can look at what can be done for extra speedups on top of it, but there's no point starting from behind On Fri, 12 Jun 2026 at 22:29, Daniel Weeks <[email protected]> wrote: > I think there are few different places we could explore improving IO > performance. > > Some options are specific to parquet, but I'm not convinced that's the > right path. > > There are some considerations as well including how we handle > encryption/metrics/etc at the FileIO implementation layer, but I think this > is a good area to invest to improve performance. > > -Dan > > > > On Fri, Jun 12, 2026 at 12:43 PM Russell Spitzer < > [email protected]> wrote: > >> I think this is a really critical idea for our work on the Root Manfiest >> as well as our indexing work. >> We are trying to minimize our latency in V4 and performing serial get >> requests on a small file is killer. >> >> I am interested in an Iceberg First solution but it does seem like >> Parquet java itself >> could use this (or have new apis to support) this pattern more expliclty >> like the Rust and C++ implementations. >> >> On Mon, Jun 8, 2026 at 4:00 PM Varun Lakhyani <[email protected]> >> wrote: >> >>> Hello everyone, >>> >>> I would like to discuss an optimization for Iceberg's Parquet read path, >>> specifically around reducing S3 GET requests for small file workloads - >>> Root Manifest, Datafiles, and small file compaction. >>> >>> *Problem* >>> The current Iceberg flow for Spark readers uses parquet-mr. For each >>> FileScanTask, it issues 3 GET requests: >>> >>> 1. Footer size discovery - 1 GET reads the last 8 bytes of the >>> Parquet file to find the actual footer size (this.currentIterator = >>> open(currentTask) in BaseReader.next) >>> 2. Footer fetch - 1 GET reads the footer (this.currentIterator = >>> open(currentTask) in BaseReader.next) >>> 3. Row group fetch - 1 GET per row group to fetch actual data >>> (this.current = currentIterator.next() in BaseReader.next) >>> >>> >>> *Background* - arrows-rs (parquet rust implementation) >>> arrow-rs already addresses the first two calls via >>> `with_footer_size_hint`. It fetches a size hint from the bottom of the file >>> containing the actual footer size - if the footer already falls within that >>> fetched range, 1 GET is eliminated. if not, a second GET fetches the >>> footer. DataFusion builds on this today. >>> For our use case, we can go further: since the files are small, instead >>> of a hint we can fetch the whole file at once in a single GET - no memory >>> concern in parquet-mr - eliminating all 3 calls entirely. >>> As the number of files grows, footer request time starts dominating over >>> actual data request time - clearly visible in benchmarks below. >>> >>> *Two Approaches* >>> >>> 1. Implement directly in Iceberg - I have a high-level PR for this >>> implementation - complete workaround in Iceberg codebase. ( >>> https://github.com/apache/iceberg/pull/16729) >>> 2. Fix upstream in parquet-mr - The architecturally correct path: >>> add this functionality to parquet-mr itself and use it entirely, >>> mirroring >>> what the Rust implementation does natively. >>> >>> >>> *JMH Benchmark Results* (20M total rows, S3, 2 warmup + 5 measurement >>> iterations) >>> Combining S3 GET requests alone gives 60-65% improvement, with further >>> gains possible by parallelising them. >>> >>> [image: image.png] >>> >>> >>> As focus shifts towards Root Manifest, Datafiles in Parquet, and >>> multiple small file requirements, a dedicated effort here seems worth >>> pursuing. >>> I would be happy to hear any thoughts on this. Points to discuss are >>> which approach seems more convincing - Iceberg implementation or upstream >>> parquet-mr implementation and further thoughts on the gaps between >>> parquet-mr and arrow-rs specifically around getting footer. >>> >>> [1] PR for high level implementation - >>> https://github.com/apache/iceberg/pull/16729 >>> -- >>> -- >>> Lakhyani Varun >>> Indian Institute of Technology Roorkee >>> Contact: +91 96246 46174 >>> >>>
