zjw1111 opened a new pull request, #554:
URL: https://github.com/apache/paimon-cpp/pull/554

   ### Purpose
   
   Continue the Vortex predicate pushdown and selective-read work started by 
@zhouyc-ali. This branch preserves the original commits through `a2631bb` and 
adds the follow-up changes in a separate commit. [Original PR 
#471](https://github.com/apache/paimon-cpp/pull/471)
   
   Supersedes #553 with identical source contents. Related issue #319.
   
   - Push supported predicates, projections, and bitmap selections into Vortex 
scans while preserving physical file row IDs. Fall back to candidate rows when 
a predicate cannot be pushed safely.
   - Make membership limits and I/O coalescing configurable per reader, and 
bound Arrow batch materialization before export, including nested values.
   - Normalize timezone metadata for new `TIMESTAMP_LTZ` columns to UTC while 
preserving epoch values. Include the dependency patches in Apache RAT checks.
   
   ### Tests
   
   - Vortex predicate converter tests cover comparisons, null checks, `IN` and 
`NOT IN`, nested `AND` and `OR`, and fallback for unsupported values and 
oversized lists.
   - Vortex file-format tests cover bitmap selections, physical row IDs, 
projections, reader reconfiguration, a user column named `_row_idx`, I/O 
coalescing, and callback and Arrow batch lifetimes.
   - Vortex table-read integration tests cover late materialization and the 
exact candidate and filtered results when an oversized `NOT IN` list falls 
back. Java fixture tests cover cross-language reads and `TIMESTAMP_LTZ` across 
timezones.
   
   ### API and Format
   
   Vortex readers accept `vortex.read.in-pushdown.max-values`, 
`vortex.read.io-coalesce.max-gap`, and `vortex.read.io-coalesce.max-size`. The 
private Vortex FFI adds scan and scalar capabilities. New Vortex files store 
`TIMESTAMP_LTZ` timezone metadata in UTC and preserve the stored epoch values. 
Public C++ signatures and the table protocol remain compatible.
   
   ### Documentation
   
   Document the Vortex reader options and defaults, predicate fallback and 
residual filtering, and `TIMESTAMP_LTZ` storage in 
`docs/source/api/file_format.rst`.
   
   ### Generative AI tooling
   
   Generated-by: OpenAI Codex 0.167.0
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to