JingsongLi opened a new pull request, #1074: URL: https://github.com/apache/paimon-rust/pull/1074
### Purpose PyPaimon already accepts numeric `incremental-between-timestamp` sources. Rust rejects the option, so prepared table-source MERGE still materializes these sources in Python. Allow the existing core scan and MERGE APIs to honor this capability without adding binding APIs or dependencies. ### Implementation - Resolve numeric `(start, end]` windows using Java `AbstractDataTableScan` and `IncrementalDeltaStartingScanner.betweenTimestamps` semantics. Empty/equal/outside windows remain empty; retain the selected ending snapshot for nonempty snapshot ranges even when files are pruned or OVERWRITE contributes no records. - Reuse `IncrementalScan` AUTO: APPEND deltas without a changelog producer, physical changelogs otherwise. Preserve repeated keys and retractions; skip OVERWRITE, and do not apply endpoint deletion vectors or global indexes to historical events. - Table-source MERGE projects and reads the selected source window inside core before staging target actions. The source remains independent of the target and is materialized only once. - Validate required selectors, conflicting point-in-time options, producer values and signed 64-bit numeric syntax. Snapshot/tag incremental options and textual timestamp ranges remain outside the existing PyPaimon capability scope. Reject unservable format/engine routes instead of returning current data. - Reuse the shared manifest reader for single incremental lists, forwarding real trace counters and removing duplicate manifest-reading code. Combined changelog splits now all identify the ending snapshot, as in Java. - Keep the Python binding as a wrapper. Document the capability and update the malformed incremental-mode error expectation. ### Java references - `AbstractDataTableScan.createIncrementalStartingScanner`: numeric bounds, empty/equal windows, AUTO mode and overwrite policy. - `IncrementalDeltaStartingScanner.betweenTimestamps`: timestamp-to-snapshot bounds and ending snapshot identity. - `SnapshotManager.earlierOrEqualTimeMills`: retain the current Java equality lookup behavior, including equal commit timestamps. ### Verification - Core library: 3,789 passed, 6 ignored. - Batch incremental, extended Diff and new timestamp integration suites: 52 passed. - Python binding read/update suites: 147 passed. - Rebuilt native extension; PyPaimon related regression suites with all Native switches enabled: 1,323 passed, 5 skipped, 85 subtests passed. All six tracked update kinds exercised. - New timestamp regressions cover append/PK/data-evolution sources, raw binding options, metadata, AUTO changelog row kinds, expired prefixes, and failure before target file staging. PyPaimon adds branch and evolved field-ID MERGE coverage with Python source snapshot resolution, materialization and matching prohibited. - CI command passed: `cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings`; Rustfmt and diff checks passed. - Two independent verification rounds completed; repaired combined changelog snapshot identity and aligned Python's numeric parsing with Java. A companion PyPaimon PR removes its timestamp-source fallback and corrects existing empty-window/AUTO parsing semantics. Native CI continues to use Rust main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
