SteNicholas opened a new pull request, #1313:
URL: https://github.com/apache/paimon-rust/pull/1313

   ### Purpose
   
   Linked issue: close #807
   
   `StreamTableScan` could only start from the latest full state: 
`scan.mode=latest` and `from-snapshot-full` were rejected, while 
`from-snapshot` and `from-timestamp` were silently ignored, so a reader asking 
for the changes since snapshot N got the latest full state instead. This PR 
resolves the starting position from `scan.mode` as Java 
`AbstractDataTableScan#createStartingScanner` does in streaming mode.
   
   ### Brief change log
   
   - **Startup modes** (`stream_table_scan.rs`), used only when the scan has 
neither a checkpoint nor consumer progress:
     - `latest`: no initial read; follow-ups after the latest snapshot, or from 
snapshot 1 on an empty table.
     - `from-snapshot`: follow-ups from `max(scan.snapshot-id, earliest)`. The 
id after the latest snapshot waits for the next commit; a larger id fails the 
existing range check, as in Java `NextSnapshotFetcher`.
     - `from-timestamp`: follow-ups from the first snapshot committed at or 
after the timestamp, or from the next commit when every snapshot is older.
     - `from-snapshot-full`: the full state of `max(scan.snapshot-id, 
earliest)`, then follow-ups. An empty table waits for its first commit; 
otherwise a missing snapshot fails with `SnapshotNotExist`.
     - An unset or `default` `scan.mode` resolves as Java 
`CoreOptions#startupMode`.
   - **Row kinds**: a full state holds the merged live rows of its snapshot as 
`+I`, without replaying earlier `+U`/`-D` events; follow-ups keep physical row 
kinds. With `changelog-producer` `none` or `input`, the full state merges every 
level, including uncompacted level-0 files of DV tables; `lookup` and 
`full-compaction` keep their level filters.
   - **Explicit errors instead of silent fallbacks** at startup: tag, watermark 
and `scan.version` selectors, incremental reads, and 
`scan.file-creation-time-millis` / `scan.creation-time-millis`, even next to 
another selector. Every plan still validates the scan options, so an invalid 
configuration (e.g. `from-snapshot-full` without `scan.snapshot-id`) also fails 
after a restore.
   - **Logging**: the start chosen from `scan.mode` or consumer progress is 
logged at INFO.
   - **`SnapshotManager::first_id_at_or_after_time_millis`**: unlike 
`later_or_equal_time_millis`, it returns the first of several snapshots sharing 
a commit time, and unlike Java's `earlierThanTimeMills + 1`, it skips ids 
missing from the listing instead of waiting on them.
   - **Python binding**: `Table.new_stream_read_builder(options)`. Unlike 
`new_read_builder`, which resolves options as batch time travel and fails when 
they select no snapshot, it only merges them, so a stream can start from the 
next commit, an expired snapshot, an early timestamp, or an empty table. It 
rejects switching branches, which merging cannot do.
   
   ### Tests
   
   - `snapshot_manager.rs`: 
`test_first_id_at_or_after_time_millis_keeps_ties_and_skips_holes`.
   - `tests/stream_table_scan_test.rs`:
     - `startup_modes_wait_on_an_empty_table`
     - `latest_and_from_snapshot_start_without_an_initial_state` (producers 
`none` / `input`, ids below the earliest, the next commit, and beyond the 
latest)
     - `from_timestamp_starts_at_the_first_snapshot_at_or_after_it` (shared 
commit times, timestamps before and after every snapshot, a snapshot-id hole)
     - `from_snapshot_full_keeps_row_kinds_across_restore` (DV and non-DV 
tables, uncompacted level 0)
     - `initial_lookup_and_full_compaction_select_java_levels_only_once` 
(extended with every startup mode for `lookup` / `full-compaction`)
     - `consumer_progress_overrides_the_startup_mode`
     - `startup_options_fail_at_startup_and_invalid_ones_also_on_restore`
     - `append_table_streams_from_each_startup_mode`
   - `bindings/python/tests/test_stream_table_scan.py`: 
`new_stream_read_builder` starting at the next commit, an early timestamp, an 
expired snapshot and an empty table, and rejecting a branch switch.
   
   ### API and Format
   
   - `StreamTableScan` now honors `scan.mode` `latest`, `from-snapshot`, 
`from-timestamp` and `from-snapshot-full`. Tables that set `scan.snapshot-id` 
or a timestamp without a checkpoint now start there instead of at the latest 
full state, and the unsupported startup options above now fail instead of being 
ignored.
   - New Python API: `Table.new_stream_read_builder`.
   - No storage format change.
   
   ### Documentation
   
   - New `docs/src/streaming-reading.md`: read loop, checkpoints, startup 
modes, row kinds and limitations; added to the `mkdocs.yml` nav.
   - `docs/src/python-binding.md`: continuous reads with 
`new_stream_read_builder`.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to