JingsongLi opened a new pull request, #841: URL: https://github.com/apache/paimon-rust/pull/841
### Purpose Follow-up to #839. Align incremental planning and `scan.version` resolution with Java so the Python native planner can preserve the same read contract. An APPEND window containing `(id=1, value=10)` followed by `(id=1, value=20)` currently reads only the newer value through the combined plan. Java packs this window as a batch but reads both physical events, including retracts. The current Rust planner also attaches endpoint deletion vectors and can prune historical files using endpoint global indexes. ### Brief change log - Carry Java's existing `isStreaming` flag through `DataSplit`, the binary codec and Python pickle payloads. Both combined and per-commit incremental plans retain L0 and read physical events without merging versions or dropping retracts. - Keep endpoint DVs and automatic global indexes out of incremental planning. Preserve explicit row selections and conservative PK stats/limit handling. Readers still honor DVs explicitly present in supplied splits. - Reject manifest DELETE entries in APPEND deltas instead of reconciling them into snapshot state; preserve Java's batch/per-commit split packing distinction. - Adapt `scan.version` before selector validation: tags take precedence, same-kind selectors are overwritten, and different-kind conflicts still fail. ### Tests - Rust table core suite: 1,279 passed, 1 ignored. - Incremental batch and audit-log integration suites: 56 passed. - Python binding `test_read.py`: 79 passed. - End-to-end validation with PyPaimon and a local wheel built from this branch, covering versions, retracts, first-row, DV, filters/limits, DE positions and split compatibility. - `cargo clippy --locked -p paimon --all-targets -- -D warnings` and `cargo fmt --all -- --check`. ### API and Format No Java binary format version changes: the existing streaming bit is preserved, and batch golden bytes remain unchanged. Missing `is_streaming` fields in older JSON payloads default to false. Python adds `Split.is_streaming()` and `serialize(*, allow_streaming=False)`. Streaming serialization requires `allow_streaming=True`; older PyPaimon decoders discard the bit, so the default rejection lets their native adapter fall back safely. Batch serialization remains unchanged. ### Documentation Updated incremental-reading and Python-binding documentation and type stubs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
