lucasfang opened a new issue, #512:
URL: https://github.com/apache/paimon-cpp/issues/512

   ## Search before asking
   
   - [x] I searched the existing issues. This is the independent-cache/warmup 
follow-up described in #419, rather than another change to the lookahead window.
   
   ## Motivation
   
   Warmup and the shared read-ahead cache currently depend on batch prefetch. 
`DataFileReaderFactory` only creates the cache when prefetch is enabled, and 
`DelegatingPrefetchReader::Warmup()` skips a file when the format or adaptive 
strategy disables prefetch.
   
   As a result, this configuration cannot warm the next file:
   
   ```cpp
   builder.EnablePrefetch(false)
       .SetReadAheadCacheEnabled(true)
       .SetWarmupLevel(WarmupLevel::RAW);
   ```
   
   These are already the default settings, but the cache and RAW warmup are 
ineffective on the synchronous read path. This also limits the 
append-compaction warmup introduced in #419 when adaptive prefetch declines a 
file.
   
   ## Solution
   
   For Parquet and ORC, make byte caching, early warmup, and parallel batch 
prefetch independent:
   
   - Share cache creation, byte-range registration, late-materialization 
callbacks, I/O metrics and cache lifecycle between synchronous and parallel 
readers.
   - Keep synchronous reads to one format reader without requiring a prefetch 
executor or queue configuration.
   - Support RAW warmup through the cache; with the cache disabled it remains a 
no-op.
   - Support DECODED warmup without prefetch by preparing only the first batch 
in the background, then reading subsequent batches synchronously. It also works 
without the cache.
   - Keep warmup active when the format or adaptive strategy declines prefetch.
   - Disable Parquet's own pre-buffering whenever the shared cache is active, 
including synchronous reads.
   
   ## Anything else?
   
   Keep the existing public API, configuration defaults, storage format and 
lookahead window. With this change, the default cache-enabled/RAW configuration 
takes effect on synchronous Parquet and ORC reads. NONE disables early warmup 
while retaining caching during normal reads.
   
   The existing costs still apply: warming can fetch bytes for a file never 
consumed, Close waits for in-flight work, and RAW only overlaps I/O when the 
filesystem's ReadAsync is asynchronous. Other format plugins retain their plain 
read path.
   
   Validation should cover cache on/off, all warmup levels, explicit and 
adaptive prefetch disablement, late materialization, filtering/bitmap and file 
row IDs, repeated warmup, schema reset, deferred failures, and early close.
   
   ## Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   
   Generated-by: OpenAI Codex (codex-cli 0.166.0)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to