JingsongLi opened a new pull request, #9825: URL: https://github.com/apache/paimon/pull/9825
### Purpose Use the planning APIs from apache/paimon-rust#831 (merged) to support native incremental and distributed scans in PyPaimon. Timestamp windows combine APPEND deltas into one plan so primary-key versions across commits are merged together; the ending snapshot supplies metadata and deletion vectors even when it contributes no APPEND files. - Add native append slices/shards, primary-key bucket shards, and Data Evolution row-position selection. Assign DE positions before explicit/global-index range pruning, and apply limits after distribution. - Preserve snapshot IDs for empty plans and explain output, validate branch resolution, and fall back to Python when the installed binding cannot support the scan. - Honor `index-file-in-data-file-dir` for deletion-vector writes, reads, and abort cleanup. Preserve explicit paths and legacy Python index locations, handle partition-directory differences, and retain multiple DV index files in one bucket. - Keep overlapping floating-point key bounds together during split planning, including conservative grouping for NaN bounds. The native planner remains optional and disabled by default. The README documents supported scans, binding requirements, and fallback behavior. ### Tests Passed locally with Python 3.11 and a newly built Rust wheel whose core and Python-binding sources match the merged Rust main: - Planner, adapter compatibility, incremental, distribution, key-bound, and deletion-vector path regressions: 165 passed. - Reader, parallel reader, and Data Evolution regressions: 174 passed in each of the Python and native planning modes; 9 Lance cases deselected in each mode. The native runs exercised 147 native plans, including ORC reads. - Flake8 and Python 3.6 grammar checks on all 24 changed Python files; `git diff --check`. The full Python dependency/version matrix and Java integration suite were not run locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
