hutiefang76 opened a new issue, #12531: URL: https://github.com/apache/seatunnel/issues/12531
DuckLake batch reads through JDBC currently need the same `SNAPSHOT_VERSION` in each worker's attach script to provide a consistent view. That manual path is covered in #12506. An automatic option would avoid selecting and distributing the version by hand, including when tables have not yet been split at checkpoint time. I'd like to add `ducklake_snapshot_auto`, defaulting to `false`, in the existing Jdbc Source: - Select one version per DuckLake catalog before schema discovery. Catalog names differing only in case share the selection. - Use the selected version for schema discovery, split bounds and reader SQL. - Keep the frozen queries and schemas in enumerator state for pending tables. Reader splits carry the pinned SQL and expected row type, and reject incompatible restored schemas before emitting records. - Fail on an unavailable snapshot instead of falling back to current data. Existing checkpoints without snapshot state cannot have the option enabled during restore. The initial scope is literal `table_path` / `table_list` selections. Custom `query` and regex selections are rejected when enabled; there is no SQL rewriting or modification of credential-bearing initialization scripts. Parallel splitting remains available through `partition_column`. Different catalogs retain independent snapshots. This does not add a change feed or Sink deduplication. I have a [working prototype](https://github.com/hutiefang76/seatunnel/commit/c5482e751358e74c51b9835abb23497e8325ae92) on top of #12506; identifier escaping in #12501 is another integration dependency. JDK 8 and 17 each passed 44 targeted checks (one separate PostgreSQL/S3 conditional test skipped), and MarkdownTest passed 4/4. The actual DuckLake test covers two tables, three readers, serialized pending-table recovery, and delete/append/schema changes after snapshot selection. Pinned reads return the original 20 rows; the default live-read control returns 15 rows including the later insert. I also extended the existing PostgreSQL IT to check rejection on a non-DuckDB source. Its full build/run is still pending because the local E2E starter dependencies are not available, so I am not counting it as passed. Does this opt-in scope fit the existing JDBC connector, or would you prefer a dialect SPI for snapshot planning? I'll keep it separate from the pending routing and Sink changes and rebase before opening the final PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
