Hi Boto, Thanks for proposing FLIP-615.
Based on what you've shared, I'd like to share a couple of thoughts. 1. It looks like when there's a composite primary key, only the first column is used[1] — could this lead to skew in that case? 2. Before making ConnectionProvider exclusive to FLIP-615, what do you think about first designing it as a shared abstraction that FLIP-602[2] (e.g. BoundaryQuerySplitterEnumerator[3]) could also reuse? I think it'd be worth discussing the overlapping areas together as we shape the abstraction. Thanks, Chanhae --- [1] https://github.com/apache/flink-connector-jdbc/pull/209/files#diff-45bca7f7ba3ee07958ab47d8a7affa8e037c7b4da1f55808a61abe7238cf9e03R261 [2] https://cwiki.apache.org/confluence/spaces/FLINK/pages/440304839/FLIP-602+Extended+JDBC+Scan+Partitioning+Options+for+Efficient+Data+Ingestion [3] https://github.com/apache/flink-connector-jdbc/pull/234 On Tue, Sep 29, 2026 at 6:08 AM Joao Boto <[email protected]> wrote: > Hi all, > > I'd like to start a discussion on FLINK-40108 [1], for which I've drafted a > FLIP introducing snapshot split enumerators for the JDBC connector's > DataStream source. > > Today, making a parallel JDBC read correct and balanced is entirely the > user's job: hand-written chunking SQL and parameter providers in the > DataStream API, and a numeric partition column with explicit bounds in the > Table/SQL API. There is also no cross-split consistency, each reader opens > its own transaction, so rows can move between chunk boundaries while a scan > is running, and enumeration progress is not checkpointed, so a failure > re-runs the scan from scratch. > > The FLIP proposes two SplitterEnumerator [2] implementations on JdbcSource, > plus the provider SPI they need: > 1. TableSplitterEnumerator — reads one table as primary-key ranged chunks > derived automatically from database metadata (keyset pagination over the PK > index, no user SQL, bounded enumerator memory). > 2. DatabaseSplitterEnumerator — discovers all tables of a schema (or an > allowlist) and emits chunked splits across them from a single source. > 3. Global snapshot consistency — all splits of one enumeration run read the > same database snapshot, implemented for PostgreSQL via pg_export_snapshot() > / SET TRANSACTION SNAPSHOT, with a dialect-neutral, capability-gated SPI in > core so other databases can follow. Unlike physical-id (ctid/ROWID) > partitioning, this remains consistent under concurrent DML. > 4. Resumable enumeration — per-table progress is checkpointed and > confirmed-split staging prevents split loss; a restored run resumes at its > cursor instead of re-enumerating, with at-least-once split delivery. > > Existing behavior is unchanged: everything is additive (new classes and > default interface methods), with a serializer version bump (v0 checkpoints > remain restorable) and one new dependency (HikariCP) that motivated this > FLIP per the dependency-change policy [3]. > > Full design: [4] > Implementation (mostly complete, happy to align details with the > discussion): https://github.com/apache/flink-connector-jdbc/pull/209 > > FLIP-602 [5] addresses partitioning strategy for the Table/SQL scan; this > FLIP is complementary and covers metadata-driven discovery, snapshot > consistency, and resumability at the DataStream level. > > Feedback and questions are very welcome. > > Thanks, > João Boto > > 1 https://issues.apache.org/jira/browse/FLINK-40108 > 2 https://issues.apache.org/jira/browse/FLINK-38733 > 3 https://github.com/apache/flink-connector-jdbc/commit/de12859 > 4 > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/451975192/FLIP-615+Snapshot+Splitters+for+Table+and+Database+Ingestion > 5 > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-602%3A+Extended+JDBC+Scan+Partitioning+Options+for+Efficient+Data+Ingestion >
