zjw1111 opened a new pull request, #554: URL: https://github.com/apache/paimon-cpp/pull/554
### Purpose Continue the Vortex predicate pushdown and selective-read work started by @zhouyc-ali. This branch preserves the original commits through `a2631bb` and adds the follow-up changes in a separate commit. [Original PR #471](https://github.com/apache/paimon-cpp/pull/471) Supersedes #553 with identical source contents. Related issue #319. - Push supported predicates, projections, and bitmap selections into Vortex scans while preserving physical file row IDs. Fall back to candidate rows when a predicate cannot be pushed safely. - Make membership limits and I/O coalescing configurable per reader, and bound Arrow batch materialization before export, including nested values. - Normalize timezone metadata for new `TIMESTAMP_LTZ` columns to UTC while preserving epoch values. Include the dependency patches in Apache RAT checks. ### Tests - Vortex predicate converter tests cover comparisons, null checks, `IN` and `NOT IN`, nested `AND` and `OR`, and fallback for unsupported values and oversized lists. - Vortex file-format tests cover bitmap selections, physical row IDs, projections, reader reconfiguration, a user column named `_row_idx`, I/O coalescing, and callback and Arrow batch lifetimes. - Vortex table-read integration tests cover late materialization and the exact candidate and filtered results when an oversized `NOT IN` list falls back. Java fixture tests cover cross-language reads and `TIMESTAMP_LTZ` across timezones. ### API and Format Vortex readers accept `vortex.read.in-pushdown.max-values`, `vortex.read.io-coalesce.max-gap`, and `vortex.read.io-coalesce.max-size`. The private Vortex FFI adds scan and scalar capabilities. New Vortex files store `TIMESTAMP_LTZ` timezone metadata in UTC and preserve the stored epoch values. Public C++ signatures and the table protocol remain compatible. ### Documentation Document the Vortex reader options and defaults, predicate fallback and residual filtering, and `TIMESTAMP_LTZ` storage in `docs/source/api/file_format.rst`. ### Generative AI tooling Generated-by: OpenAI Codex 0.167.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
