lucasfang opened a new pull request, #209:
URL: https://github.com/apache/paimon-cpp/pull/209
### Purpose
Linked issue: close #xxx
The framework-level prefetch previously only had working `PreBufferRange()`
support for ORC; Parquet reads could not feed the shared read-ahead cache.
This change completes the Parquet side and makes the cache observable:
1. **Parquet pre-buffer ranges**: `ParquetFileBatchReader::PreBufferRange()`
is implemented via `FileReaderWrapper::GetPreBufferRanges()`, which
computes
column-chunk ranges (dictionary-page aware) for fully-matched row groups
and
page-level byte ranges for page-filtered row groups.
2. **Read-ahead cache integration**: `PrefetchFileBatchReaderImpl` creates a
shared `ReadAheadCache` per file reader according to `PrefetchCacheMode`
(`ALWAYS` / `EXCLUDE_PREDICATE` / `EXCLUDE_BITMAP` /
`EXCLUDE_BITMAP_OR_PREDICATE` / `NEVER`). The workloop registers the
ranges
(`Init`) and dispatches prefetch (`Warmup`) ahead of the readers;
positional
reads issued through `CacheInputStream` are served from the cache, with
in-flight fetches awaited instead of re-fetched. `CacheConfig` defaults
are
aligned with the reader request granularity (32 MiB range limit / 8 KiB
hole
limit / 256 MiB pre-buffer window / 1 GiB buffer limit).
3. **Single prefetch layer in ALWAYS mode**: in `ALWAYS` mode the
reader-level
(arrow-internal) `pre_buffer` is disabled through
`ReaderBuilder::WithPreBufferEnabled(false)`, so arrow's `ReadRangeCache`
does not hold a second in-memory copy of the same column chunks. Other
cache
modes keep the reader's own prebuffer because the per-file
predicate/bitmap
is not known at split-read level.
4. **Cache metrics surfaced to users**: hit/miss counters
(`read-ahead-cache.read.hits` / `.hit-bytes` / `.misses` / `.miss-bytes`)
are merged into `BatchReader::GetReaderMetrics()`. This includes two
fixes:
- `ConcatBatchReader` closes file readers at EOF, and `Close()` used to
reset the cache counters, so metrics read after the scan were always
zero.
The full `Reset()` now only happens on the reader-reuse entry points
(`SetReadSchema` / `RefreshReadRanges`).
- `ReadAheadCache::ReleaseBuffers()` is added so that `Close()` still
frees
the prefetched buffers of a finished file immediately (avoiding per-file
buffer accumulation across a multi-file scan) while keeping the counters
readable.
5. **ResolveBucketMode consistency** (secondary fix bundled in):
`POSTPONE_MODE`
is now only resolved for primary-key tables; the dead `UNAWARE_BUCKET`
branch is removed and the constants are documented.
### Tests
- UT `read_ahead_cache_test.cpp`: basics, repeated-read reuse, eviction,
metrics counters, and the new `TestReleaseBuffersKeepsMetrics` (buffers are
dropped but counters survive `ReleaseBuffers()`, and `Reset()` zeroes
them).
- UT `parquet_file_batch_reader_test.cpp` (+182 lines): end-to-end pre-buffer
range coverage incl. page-filtered row groups and cache-hit reads.
- UT `bucket_mode_test.cpp`: updated for the postpone/PK rule.
- IT `read_inte_test.cpp::TestReadAheadCacheMetrics`: parameterized over
parquet/orc and all cache modes; asserts the counters are present and
`hits > 0` exactly when a cache is created, absent otherwise.
### API and Format
Additive public API only, no storage format or protocol changes:
- `include/paimon/utils/read_ahead_cache.h`: `ReadAheadCacheMetrics` counter
names, `PrefetchCacheMode`, `CacheConfig` tuning knobs, and
`ReadAheadCache::ReleaseBuffers()`.
- `include/paimon/format/reader_builder.h`: new virtual
`ReaderBuilder::WithPreBufferEnabled(bool)` (no-op base default; overridden
by `ParquetReaderBuilder`).
### Documentation
No new user-facing feature flag; prefetch behavior is covered by the existing
`docs/source/user_guide/prefetch.rst`. No documentation changes required.
### Generative AI tooling
Generated-by: Qoder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
