SteNicholas opened a new pull request, #1313:
URL: https://github.com/apache/paimon-rust/pull/1313
### Purpose
Linked issue: close #807
`StreamTableScan` could only start from the latest full state:
`scan.mode=latest` and `from-snapshot-full` were rejected, while
`from-snapshot` and `from-timestamp` were silently ignored, so a reader asking
for the changes since snapshot N got the latest full state instead. This PR
resolves the starting position from `scan.mode` as Java
`AbstractDataTableScan#createStartingScanner` does in streaming mode.
### Brief change log
- **Startup modes** (`stream_table_scan.rs`), used only when the scan has
neither a checkpoint nor consumer progress:
- `latest`: no initial read; follow-ups after the latest snapshot, or from
snapshot 1 on an empty table.
- `from-snapshot`: follow-ups from `max(scan.snapshot-id, earliest)`. The
id after the latest snapshot waits for the next commit; a larger id fails the
existing range check, as in Java `NextSnapshotFetcher`.
- `from-timestamp`: follow-ups from the first snapshot committed at or
after the timestamp, or from the next commit when every snapshot is older.
- `from-snapshot-full`: the full state of `max(scan.snapshot-id,
earliest)`, then follow-ups. An empty table waits for its first commit;
otherwise a missing snapshot fails with `SnapshotNotExist`.
- An unset or `default` `scan.mode` resolves as Java
`CoreOptions#startupMode`.
- **Row kinds**: a full state holds the merged live rows of its snapshot as
`+I`, without replaying earlier `+U`/`-D` events; follow-ups keep physical row
kinds. With `changelog-producer` `none` or `input`, the full state merges every
level, including uncompacted level-0 files of DV tables; `lookup` and
`full-compaction` keep their level filters.
- **Explicit errors instead of silent fallbacks** at startup: tag, watermark
and `scan.version` selectors, incremental reads, and
`scan.file-creation-time-millis` / `scan.creation-time-millis`, even next to
another selector. Every plan still validates the scan options, so an invalid
configuration (e.g. `from-snapshot-full` without `scan.snapshot-id`) also fails
after a restore.
- **Logging**: the start chosen from `scan.mode` or consumer progress is
logged at INFO.
- **`SnapshotManager::first_id_at_or_after_time_millis`**: unlike
`later_or_equal_time_millis`, it returns the first of several snapshots sharing
a commit time, and unlike Java's `earlierThanTimeMills + 1`, it skips ids
missing from the listing instead of waiting on them.
- **Python binding**: `Table.new_stream_read_builder(options)`. Unlike
`new_read_builder`, which resolves options as batch time travel and fails when
they select no snapshot, it only merges them, so a stream can start from the
next commit, an expired snapshot, an early timestamp, or an empty table. It
rejects switching branches, which merging cannot do.
### Tests
- `snapshot_manager.rs`:
`test_first_id_at_or_after_time_millis_keeps_ties_and_skips_holes`.
- `tests/stream_table_scan_test.rs`:
- `startup_modes_wait_on_an_empty_table`
- `latest_and_from_snapshot_start_without_an_initial_state` (producers
`none` / `input`, ids below the earliest, the next commit, and beyond the
latest)
- `from_timestamp_starts_at_the_first_snapshot_at_or_after_it` (shared
commit times, timestamps before and after every snapshot, a snapshot-id hole)
- `from_snapshot_full_keeps_row_kinds_across_restore` (DV and non-DV
tables, uncompacted level 0)
- `initial_lookup_and_full_compaction_select_java_levels_only_once`
(extended with every startup mode for `lookup` / `full-compaction`)
- `consumer_progress_overrides_the_startup_mode`
- `startup_options_fail_at_startup_and_invalid_ones_also_on_restore`
- `append_table_streams_from_each_startup_mode`
- `bindings/python/tests/test_stream_table_scan.py`:
`new_stream_read_builder` starting at the next commit, an early timestamp, an
expired snapshot and an empty table, and rejecting a branch switch.
### API and Format
- `StreamTableScan` now honors `scan.mode` `latest`, `from-snapshot`,
`from-timestamp` and `from-snapshot-full`. Tables that set `scan.snapshot-id`
or a timestamp without a checkpoint now start there instead of at the latest
full state, and the unsupported startup options above now fail instead of being
ignored.
- New Python API: `Table.new_stream_read_builder`.
- No storage format change.
### Documentation
- New `docs/src/streaming-reading.md`: read loop, checkpoints, startup
modes, row kinds and limitations; added to the `mkdocs.yml` nav.
- `docs/src/python-binding.md`: continuous reads with
`new_stream_read_builder`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]