lucasfang opened a new issue, #218: URL: https://github.com/apache/paimon-cpp/issues/218
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation # [Feature] Support read-ahead cache for Parquet reads Labels: enhancement ### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation The framework-level prefetch (`PrefetchFileBatchReader` + shared read-ahead cache) currently only has a working `PreBufferRange()` implementation for ORC, so Parquet reads cannot feed the shared cache and get no latency hiding from it. Several concrete problems follow from that: 1. **Parquet reads cannot use the shared prefetch cache**: `ParquetFileBatchReader::PreBufferRange()` is unimplemented, so on high-latency storage (OSS/S3/HDFS) every column-chunk read is a synchronous round trip and sequential scans pay the full IO latency. 2. **Double prefetch / double memory**: enabling arrow's internal `pre_buffer` (`parquet.read.enable-pre-buffer`) as a workaround makes arrow's `ReadRangeCache` hold a second in-memory copy of the same column chunks, doubling memory for prefetched data. 3. **SeekToRow double-read amplification**: `FileReaderWrapper::SeekToRow()` issued before the first `Next()` rebuilds a batch reader that `PrepareForReading` immediately discards, and since the patched arrow `GetRecordBatchReader` eagerly reads every column chunk, building the reader twice doubles the requested bytes. 4. **No observability / configurability**: there are no hit/miss or prefetch IO counters for the cache, and users cannot switch cache prefetch off for predicate/bitmap-index queries where prefetching is wasteful. ### Solution 1. **Parquet pre-buffer ranges**: implement `ParquetFileBatchReader::PreBufferRange()` via `FileReaderWrapper::GetPreBufferRanges()`, computing dictionary-page-aware column-chunk ranges for fully-matched row groups and page-level byte ranges for page-filtered row groups. Unlike the arrow-internal PreBuffer path, this also covers row groups excluded by read-range dispatch, because the shared prefetch cache must serve data consumed by all sub-readers. 2. **Read-ahead cache integration**: `PrefetchFileBatchReaderImpl` creates a shared `ReadAheadCache` per file reader, registers ranges via `Init()` and dispatches prefetch via `Warmup()` ahead of the readers; positional reads issued through `CacheInputStream` are served from the cache via a copy-out `Read()` API, with in-flight fetches awaited instead of re-fetched. 3. **Public config-only API**: keep the cache implementation internal (`src/paimon/common/utils/read_ahead_cache.h`) and expose only `PrefetchCacheMode` (`ALWAYS` / `EXCLUDE_PREDICATE` / `EXCLUDE_BITMAP` / `EXCLUDE_BITMAP_OR_PREDICATE` / `NEVER`) and `CacheConfig` tuning knobs (buffer/range/hole/pre-buffer limits) in a new public header `include/paimon/utils/prefetch_cache_config.h`, consumed by `ReadContextBuilder::SetPrefetchCacheMode()` / `WithCacheConfig()`. 4. **Single prefetch layer**: default `parquet.read.enable-pre-buffer` to false so arrow's `ReadRangeCache` does not hold a second in-memory copy of column chunks already fetched by the shared cache; the option stays available to restore the arrow-internal prebuffer when the read-ahead cache is disabled. 5. **SeekToRow fix**: record a pre-initialization seek target in `FileReaderWrapper` (`pending_start_idx_`) and consume it in the deferred `PrepareForReading`, so no batch reader is built twice and no extra IO is issued. 6. **Metrics**: expose read-ahead cache counters (`read-ahead-cache.read.count/bytes`, `.read.hits/hit-bytes/misses/miss-bytes`, `.io.count/bytes`) merged into `BatchReader::GetReaderMetrics()`, with cache buffers released on reader `Close()` (`ReleaseBuffers()`) while keeping counters readable for post-scan metric aggregation. ### Anything else? The cache design is adapted from the Apache ORC C++ reader cache (https://github.com/apache/orc/blob/main/c%2B%2B/src/io/Cache.hh). A bundled secondary fix aligns `ResolveBucketMode` with Java Paimon: `POSTPONE_MODE` is only resolved for primary-key tables. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! ### Solution _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
