lucasfang opened a new pull request, #513:
URL: https://github.com/apache/paimon-cpp/pull/513

   ### Purpose
   
   Linked issue: Closes #512
   
   With batch prefetch disabled, Parquet and ORC previously bypassed the shared 
read-ahead cache and could not warm the next file. Warmup also stopped when the 
format or adaptive strategy declined prefetch.
   
   This change makes the existing settings independent. The default 
`EnablePrefetch(false)`, cache-enabled, RAW configuration now caches and warms 
synchronous reads. RAW only fetches bytes; DECODED prepares one batch in the 
background before continuing synchronously. Active prefetch keeps its existing 
parallel decode loop.
   
   - Extract shared cache creation, range registration, late-materialization 
callbacks, deferred errors and I/O metrics into `FileReadAhead`.
   - Add a synchronous warmup wrapper that joins/discards pending work before 
schema reset, close or destruction; no prefetch executor or queue settings are 
required.
   - Route adaptive fallback through the same wrapper and let an active shared 
cache replace Parquet's own pre-buffering.
   - Preserve plain reads for other format plugins and account for the cache's 
separate stream in the main-reader reuse regression test.
   
   ### Tests
   
   Local Debug build with tests enabled:
   
   ```bash
   cmake --build build --target paimon-common-test paimon-core-test 
paimon-parquet-format-test paimon-orc-format-test -j 32
   ./build/debug/paimon-common-test
   ./build/debug/paimon-core-test
   ./build/debug/paimon-parquet-format-test
   ./build/debug/paimon-orc-format-test
   git diff --check
   ```
   
   All four complete test executables passed: common 1,864 passed / 2 
conditional cache-disabled skips; core 2,622 passed; Parquet 240 passed; ORC 
120 passed. Total: 4,846 passed, 2 skipped, 0 failures.
   
   Regression coverage includes real Parquet/ORC combinations of 
prefetch/cache/warmup settings, adaptive and format fallback, late 
materialization with predicates and bitmaps, file row IDs, multiple files, 
one-batch background preparation, repeated warmup, schema reset, deferred 
failures and unconsumed close/destruction. Controlled asynchronous streams and 
completion signals verify warmup behavior without timing-based ordering 
assertions.
   
   Read-only clang-format 20.1.8, cmake-format and whitespace checks passed, as 
did applicable pre-commit large-file, spelling, Sphinx and C++ lint hooks. The 
rewriting format/whitespace hooks were replaced by those read-only checks. 
Repository-wide `unittest`, integration executables and sanitizer builds were 
not run locally.
   
   ### API and Format
   
   No public signature, enum value, configuration default, storage format or 
protocol change. Public API comments now describe independent operation.
   
   Behavior changes intentionally: synchronous Parquet/ORC reads now honor the 
already-enabled read-ahead cache and RAW default. DECODED without prefetch 
buffers one batch; RAW requires the cache. Cache/I/O metrics remain available 
independently of batch prefetch, and synchronous warmup reports prefetch 
disabled.
   
   ### Documentation
   
   Update the prefetch/warmup guide, I/O metrics guide and API comments. Retain 
documentation of memory costs, early-stop cleanup, and RAW's 
synchronous-filesystem limitation.
   
   ### Generative AI tooling
   
   Generated-by: OpenAI Codex (codex-cli 0.166.0)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to