zjw1111 opened a new pull request, #553:
URL: https://github.com/apache/paimon-cpp/pull/553

   ### Purpose
   
   Continue the Vortex predicate pushdown and selective-read work started by 
@zhouyc-ali. [Original pull request 
#471](https://github.com/apache/paimon-cpp/pull/471)
   
   Related issue: #319.
   
   - Push supported predicates, projections, and bitmap selections into Vortex 
scans while preserving physical file row IDs. Fall back to candidate rows when 
a predicate cannot be pushed safely.
   - Make membership limits and I/O coalescing configurable per reader. Bound 
Arrow batch materialization before export, including nested values.
   - Normalize timezone metadata for newly written TIMESTAMP_LTZ columns to UTC 
while preserving their epoch values. Add ASF headers to the dependency patches 
and include them in RAT checks.
   
   ### Tests
   
   - Vortex predicate converter tests cover comparisons, null checks, IN and 
NOT IN, nested AND/OR expressions, and safe fallback for unsupported values and 
oversized lists.
   - Vortex file reader tests cover sparse and dense selections, physical row 
IDs, projection and reader reconfiguration, user columns named `_row_idx`, and 
callback I/O and Arrow batch ownership.
   - Rust FFI tests cover bounded export of nested arrays, date and timestamp 
scalar bounds, I/O coalescing, and callback error and cancellation handling.
   - Vortex table-read integration tests cover late materialization with native 
filtering and the complete candidate and exact results when an oversized NOT IN 
list falls back.
   
   ### API and Format
   
   Vortex readers accept `vortex.read.in-pushdown.max-values`, 
`vortex.read.io-coalesce.max-gap`, and `vortex.read.io-coalesce.max-size`. The 
private Vortex FFI adds scan and scalar capabilities. New TIMESTAMP_LTZ files 
record UTC timezone metadata and preserve the stored epoch values.
   
   ### Documentation
   
   Document the Vortex reader options and defaults, predicate fallback rules, 
residual filtering, and TIMESTAMP_LTZ storage in 
`docs/source/api/file_format.rst`.
   
   ### Generative AI tooling
   
   Generated-by: OpenAI Codex 0.167.0
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to