plusplusjiajia opened a new pull request, #10857: URL: https://github.com/apache/paimon/pull/10857
### Purpose PyPaimon only recognized `incremental-between-timestamp`, so batch reads with `incremental-between` set to two snapshot ids or two tags fell back to a full read of the latest snapshot, and `incremental-between-scan-mode` and `incremental-to-auto-tag` were ignored. As in Java `AbstractDataTableScan`: - `incremental-between` reads the changes between two snapshot ids or two tags, start exclusive and end inclusive, with Java's validation of the range. Tags are resolved before snapshot ids, and branch tables use their own history. - `incremental-between-scan-mode` is honored for both `incremental-between` and `incremental-between-timestamp`: `delta` and `changelog` read the delta or changelog files, and `auto` picks one by `changelog-producer`. The `diff` mode, the default diff between two tags (without `incremental-between-tag-to-snapshot`) and `incremental-to-auto-tag` are not supported and now raise `NotImplementedError` instead of being ignored. Code that pins a snapshot or rejects incremental scans (snapshot copies, temporal alignment, window datasets and Ray joins) now treats `incremental-between` and `incremental-to-auto-tag` like `incremental-between-timestamp`, and native planning leaves these reads to the Python scanner. ### Tests - `native_plan_incremental_test.py`: snapshot id and tag ranges on append and primary-key tables, range validation, expired and empty histories, tag resolution order and branch history, the scan modes, and the unsupported options. - `table_scan_mode_test.py`, `file_store_table_test.py`, `native_plan_test.py`, `multimodal_temporal_test.py` and `table_write_test.py`: option validation, snapshot pinning, native fallback, temporal alignment and writes through an incremental copy. - 59 of the new test cases fail on master; the write test guards existing behavior. The related tests pass locally: 376 passed, with native runtime cases skipped. flake8 is clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
