varun-lakhyani opened a new pull request, #16729:
URL: https://github.com/apache/iceberg/pull/16729
For small file workloads (compaction, manifests, small data files), footer
request latency grows linearly with file count and becomes a scaling
bottleneck.
**Solution**
- Introduce SingleFetchInputFile, an InputFile decorator in
org.apache.iceberg.io. When a file's size is at or below
read.single-fetch-threshold-bytes, newStream() fetches the entire file in one
GET and returns a SingleFetchInputStream backed by the downloaded bytes. All
subsequent parquet-mr reads - footer size, footer body, row group data - are
served from memory with no additional remote calls.
- Files above the threshold, or when the property is unset (0, the default),
pass through unchanged with no behavioral difference from today.
- SingleFetchInputStream implements both SeekableInputStream and
RangeReadable, preserving parquet-mr's vectorized read path exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
