Hi Sergey, David, I hope this message finds you well. I am writing to ask if you would be willing to share your thoughts on a FLIP I have been working on — Extended JDBC Scan Partitioning Options for Efficient Data Ingestion.
The proposal introduces two extensions to the JDBC connector's scan partitioning: 1. *scan.partition.use-physical-id* — For Oracle and PostgreSQL, which expose a physical row identifier (ROWID / ctid), partition bounds are derived directly from DB-native metadata. Because data is physically distributed across storage blocks, this produces naturally balanced splits without requiring any knowledge of the data distribution. 2. *scan.partition.boundary-query* — A SQL query returning ordered split points for the partition column. Unlike lower-bound / upper-bound, which divide a range into equal intervals regardless of actual distribution, this option lets users place split points where data is actually concentrated. It also extends support to string-type partition columns, which the current implementation does not handle. A discussion thread is already open on the dev mailing list, and the full draft is linked below. - JIRA: https://issues.apache.org/jira/browse/FLINK-36812 - FLIP draft: https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing - Discussion: https://lists.apache.org/thread/wxh94hhyoy4b164m9q6cxhf8jypkpwdo Any feedback or suggestions would be greatly appreciated. Best regards, ChanHae Oh ------------------------------ *From:* Chanhae Oh <[email protected]> *Sent:* Tuesday, July 7, 2026 19:09 *To:* Ryan van Huuksloot via dev <[email protected]> *Subject:* [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion Hi all, Following up on FLINK-36812 [1], I've drafted a FLIP to address the limitations of the current scan.partition.column implementation. The existing approach splits data by equal numeric intervals, which breaks down under two common scenarios: skewed data distribution and the absence of a suitable numeric column (e.g., string-typed partition keys). Two extensions are proposed: *1. Physical ID Partitioning (scan.partition.use-physical-id)* For databases that expose a physical row identifier (Oracle ROWID, PostgreSQL ctid), the connector can derive partition bounds automatically from DB-native metadata. Since rows are physically distributed across storage blocks, this produces naturally balanced splits with no user-defined partition column required. *2. Boundary Query (scan.partition.boundary-query)* A SQL query returning ordered split points for the column specified in scan.partition.column. The connector uses these values to generate range predicates, enabling distribution-aware parallel reads for any column type — including strings. scan.partition.num is required and determines how many partitions are created. This is conceptually close to Apache Sqoop's --boundary-query, extended to support string-type columns natively. Both extensions leave existing behavior unchanged. Full design: [2] Feedback and questions are welcome. Thanks, Chanhae Oh [1] https://issues.apache.org/jira/browse/FLINK-36812 [2] https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing
