lucasfang opened a new issue, #512:
URL: https://github.com/apache/paimon-cpp/issues/512
## Search before asking
- [x] I searched the existing issues. This is the independent-cache/warmup
follow-up described in #419, rather than another change to the lookahead window.
## Motivation
Warmup and the shared read-ahead cache currently depend on batch prefetch.
`DataFileReaderFactory` only creates the cache when prefetch is enabled, and
`DelegatingPrefetchReader::Warmup()` skips a file when the format or adaptive
strategy disables prefetch.
As a result, this configuration cannot warm the next file:
```cpp
builder.EnablePrefetch(false)
.SetReadAheadCacheEnabled(true)
.SetWarmupLevel(WarmupLevel::RAW);
```
These are already the default settings, but the cache and RAW warmup are
ineffective on the synchronous read path. This also limits the
append-compaction warmup introduced in #419 when adaptive prefetch declines a
file.
## Solution
For Parquet and ORC, make byte caching, early warmup, and parallel batch
prefetch independent:
- Share cache creation, byte-range registration, late-materialization
callbacks, I/O metrics and cache lifecycle between synchronous and parallel
readers.
- Keep synchronous reads to one format reader without requiring a prefetch
executor or queue configuration.
- Support RAW warmup through the cache; with the cache disabled it remains a
no-op.
- Support DECODED warmup without prefetch by preparing only the first batch
in the background, then reading subsequent batches synchronously. It also works
without the cache.
- Keep warmup active when the format or adaptive strategy declines prefetch.
- Disable Parquet's own pre-buffering whenever the shared cache is active,
including synchronous reads.
## Anything else?
Keep the existing public API, configuration defaults, storage format and
lookahead window. With this change, the default cache-enabled/RAW configuration
takes effect on synchronous Parquet and ORC reads. NONE disables early warmup
while retaining caching during normal reads.
The existing costs still apply: warming can fetch bytes for a file never
consumed, Close waits for in-flight work, and RAW only overlaps I/O when the
filesystem's ReadAsync is asynchronous. Other format plugins retain their plain
read path.
Validation should cover cache on/off, all warmup levels, explicit and
adaptive prefetch disablement, late materialization, filtering/bitmap and file
row IDs, repeated warmup, schema reset, deferred failures, and early close.
## Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Generated-by: OpenAI Codex (codex-cli 0.166.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]