lucasfang opened a new pull request, #522:
URL: https://github.com/apache/paimon-cpp/pull/522

   ### Purpose
   
   Fix two existing main-branch bugs identified in the review of #513:
   
   - [Validate byte ranges against file 
size](https://github.com/apache/paimon-cpp/pull/513#discussion_r4476202458). A 
cache for a 1 KiB file previously accepted a 32 MiB range starting at offset 
512. Validate the entire registration against the known file size before the 
coalescer splits ranges or cache state changes. The subtraction-based check 
avoids integer overflow, rejects invalid late registrations atomically, and 
preserves zero as the unknown-size sentinel.
   - [Skip payload range queries when caching is 
disabled](https://github.com/apache/paimon-cpp/pull/513#discussion_r4476202461).
 With prefetch enabled, caching disabled and `warmup=NONE`, an unnecessary LM 
payload `PreBufferRange()` failure previously failed an otherwise readable 
scan. Install the callback only when the shared cache exists.
   
   ### Tests
   
   Five new regression tests cover out-of-file and overflowing ranges, invalid 
late registrations, exact EOF boundaries, unknown file sizes, and 
cache-disabled LM reads under NONE/RAW/DECODED warmup. The LM test makes every 
range query fail and verifies all filtered keys and payload values, so 
successful reads require skipping those queries.
   
   Before the production fix, the five tests produced the expected three 
failures and two compatibility passes. After the fix, all five pass.
   
   Native Debug validation (`build-rest`, ORC/Parquet/Avro/REST enabled):
   
   ```sh
   cmake --build build-rest --target paimon-common-test 
paimon-parquet-format-test paimon-orc-format-test -j16
   ./build-rest/debug/paimon-common-test
   ./build-rest/debug/paimon-parquet-format-test
   ./build-rest/debug/paimon-orc-format-test
   pre-commit run --files src/paimon/common/utils/read_ahead_cache.cpp 
src/paimon/common/utils/read_ahead_cache.h 
src/paimon/common/utils/read_ahead_cache_test.cpp 
src/paimon/common/reader/prefetch_file_batch_reader_impl.cpp 
src/paimon/common/reader/late_materializing_file_batch_reader_test.cpp
   git diff --check
   ```
   
   - Common: 1,862 passed; two existing parameterized cases skip because their 
cache is disabled.
   - Parquet: 240 passed.
   - ORC: 120 passed.
   - Formatting, lint and whitespace checks passed for all five changed files.
   
   The all-component build was attempted but could not finish because of local 
Rust dependency/toolchain incompatibilities (Vortex with Rust 1.67, then 
Lance/ethnum with installed stable Rust 1.97). The successful native build 
disables those optional Rust formats. Repository-wide `unittest`, integration 
executables and sanitizer builds were not run.
   
   ### API and Format
   
   No changes to installed public API, storage format or protocol. Malformed 
ranges beyond a known file size now return a validation error before 
allocation/splitting; file size zero retains its existing unknown-size behavior.
   
   ### Documentation
   
   Updated the internal `ReadAheadCache` constructor and `AddRanges()` comments 
to describe validation and the unknown-size convention. No new feature or 
configuration option.
   
   ### Generative AI tooling
   
   Generated-by: OpenAI Codex (codex-cli 0.166.0)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to