Hi all, I'd like to start a discussion on FLINK-40108 [1], for which I've drafted a FLIP introducing snapshot split enumerators for the JDBC connector's DataStream source.
Today, making a parallel JDBC read correct and balanced is entirely the user's job: hand-written chunking SQL and parameter providers in the DataStream API, and a numeric partition column with explicit bounds in the Table/SQL API. There is also no cross-split consistency, each reader opens its own transaction, so rows can move between chunk boundaries while a scan is running, and enumeration progress is not checkpointed, so a failure re-runs the scan from scratch. The FLIP proposes two SplitterEnumerator [2] implementations on JdbcSource, plus the provider SPI they need: 1. TableSplitterEnumerator — reads one table as primary-key ranged chunks derived automatically from database metadata (keyset pagination over the PK index, no user SQL, bounded enumerator memory). 2. DatabaseSplitterEnumerator — discovers all tables of a schema (or an allowlist) and emits chunked splits across them from a single source. 3. Global snapshot consistency — all splits of one enumeration run read the same database snapshot, implemented for PostgreSQL via pg_export_snapshot() / SET TRANSACTION SNAPSHOT, with a dialect-neutral, capability-gated SPI in core so other databases can follow. Unlike physical-id (ctid/ROWID) partitioning, this remains consistent under concurrent DML. 4. Resumable enumeration — per-table progress is checkpointed and confirmed-split staging prevents split loss; a restored run resumes at its cursor instead of re-enumerating, with at-least-once split delivery. Existing behavior is unchanged: everything is additive (new classes and default interface methods), with a serializer version bump (v0 checkpoints remain restorable) and one new dependency (HikariCP) that motivated this FLIP per the dependency-change policy [3]. Full design: [4] Implementation (mostly complete, happy to align details with the discussion): https://github.com/apache/flink-connector-jdbc/pull/209 FLIP-602 [5] addresses partitioning strategy for the Table/SQL scan; this FLIP is complementary and covers metadata-driven discovery, snapshot consistency, and resumability at the DataStream level. Feedback and questions are very welcome. Thanks, João Boto 1 https://issues.apache.org/jira/browse/FLINK-40108 2 https://issues.apache.org/jira/browse/FLINK-38733 3 https://github.com/apache/flink-connector-jdbc/commit/de12859 4 https://cwiki.apache.org/confluence/spaces/FLINK/pages/451975192/FLIP-615+Snapshot+Splitters+for+Table+and+Database+Ingestion 5 https://cwiki.apache.org/confluence/display/FLINK/FLIP-602%3A+Extended+JDBC+Scan+Partitioning+Options+for+Efficient+Data+Ingestion
