u70b3 opened a new pull request, #677: URL: https://github.com/apache/paimon-rust/pull/677
### Purpose <!-- Linking this pull request to the issue --> Linked issue: close #676 Support watermark-based batch time travel, mirroring Java's `StaticFromWatermarkStartingScanner`: `scan.watermark` resolves the earliest snapshot whose watermark is greater than or equal to the requested value and scans it in full. Today the option is on the `validate_scan_options` blocklist, so Java-written tables carrying it cannot be read from Rust at all. ### Brief change log - `SnapshotManager::later_or_equal_watermark`: binary search over the actual snapshot id list (gap-tolerant, same pattern as `later_or_equal_time_millis`). Snapshots without a watermark are skipped — both `None` and `Some(i64::MIN)`, since Flink writers use `Long.MIN_VALUE` as the no-watermark sentinel. - `CoreOptions`: `scan.watermark` becomes a first-class `TimeTravelSelector` (mutual exclusion with the other selectors, strict i64 parsing); removed from the unsupported scan-option blocklist; accepted under `scan.mode=from-snapshot` (Java's `startupMode()` maps it to `FROM_SNAPSHOT`). - `travel_to_snapshot`: resolve the selector; no match fails at scan planning with Java's message, while `copy_with_time_travel` keeps Java's silent-fallback behavior. - `Table::copy_with_options`: changing `scan.watermark` invalidates the cached resolved snapshot. - C bindings: the `unsupported_scan_option_is_rejected` test now uses `incremental-between` as its example (`scan.watermark` is supported now). - `docs/src/sql.md`: new "By Watermark" subsection under Time Travel. Assumptions / deviations to be aware of (per the AI-assisted PR policy): - **Deliberate deviation from Java**: Java's binary search records the raw mid snapshot as a candidate, which may itself carry no watermark; this implementation returns the earliest snapshot that actually matches. For the batch full-scan use (`ScanMode.ALL`) this is equivalent — Java scans the resolved snapshot in full either way. Documented in the method's doc comment. - Watermarks are assumed non-decreasing in snapshot order (guaranteed by Flink/Java writers); the binary search relies on this. - Rust's own commit path never writes watermarks, so the new tests commit `Snapshot`s with watermarks directly through `SnapshotManager`; no write-path changes are included. ### Tests - `cargo fmt --all -- --check` - `cargo clippy --locked --all-targets -p paimon -- -D warnings` - `cargo test --locked -p paimon --lib` — 2100 passed, 0 failed (10 new watermark tests: all-sentinel aligned with Java's `testLaterOrEqualWatermark`, mixed `None`/valued watermarks, exact/between/out-of-range matches, snapshot-id gaps, silent fallback + scan-time error, selector mutual exclusion, cache invalidation) - `cargo check --locked -p paimon-datafusion` (dynamic options like `SET 'paimon.scan.watermark'` flow through the catalog with no changes needed) - `cargo test --locked -p paimon-c unsupported_scan_option` — 1 passed ### API and Format No public API or storage format changes. `SnapshotManager::later_or_equal_watermark` is a new public method; `SCAN_WATERMARK_OPTION` becomes public alongside the other selector constants. ### Documentation `docs/src/sql.md` Time Travel section gains a "By Watermark" subsection documenting the `SET 'paimon.scan.watermark'` dynamic option. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
