Hi Chanhae I fully support this proposal.
The current JDBC connector's limitation to numeric partition columns is a significant pain point in practice. String-based partitioning via scan.partition.column and the introduction of scan.partition.boundary-query (inspired by Sqoop) are practical additions that directly address real-world use cases. In particular, the scan.partition.use-physical-id option for Oracle and PostgreSQL using ROWID-based splitting is a clean approach to avoiding data skew — something that has been difficult to handle reliably with the existing strategy. Looking forward to seeing this move forward. Happy to help with review or testing if needed. Best regards, Young il Kim. 2026년 7월 8일 (수) 12:10, Chanhae Oh <[email protected]>님이 작성: > Hi Sergey, David, > > I hope this message finds you well. > I am writing to ask if you would be willing to share your thoughts on a > FLIP I have been working on — Extended JDBC Scan Partitioning Options for > Efficient Data Ingestion. > > The proposal introduces two extensions to the JDBC connector's scan > partitioning: > > 1. *scan.partition.use-physical-id* — For Oracle and PostgreSQL, which > expose a physical row identifier (ROWID / ctid), partition bounds are > derived directly from DB-native metadata. Because data is physically > distributed across storage blocks, this produces naturally balanced splits > without requiring any knowledge of the data distribution. > > 2. *scan.partition.boundary-query* — A SQL query returning ordered split > points for the partition column. Unlike lower-bound / upper-bound, which > divide a range into equal intervals regardless of actual distribution, this > option lets users place split points where data is actually concentrated. > It also extends support to string-type partition columns, which the current > implementation does not handle. > > A discussion thread is already open on the dev mailing list, and the full > draft is linked below. > > > - JIRA: https://issues.apache.org/jira/browse/FLINK-36812 > - FLIP draft: > > https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing > - Discussion: > https://lists.apache.org/thread/wxh94hhyoy4b164m9q6cxhf8jypkpwdo > > > Any feedback or suggestions would be greatly appreciated. > > Best regards, > ChanHae Oh > ------------------------------ > *From:* Chanhae Oh <[email protected]> > *Sent:* Tuesday, July 7, 2026 19:09 > *To:* Ryan van Huuksloot via dev <[email protected]> > *Subject:* [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for > Efficient Data Ingestion > > Hi all, > > Following up on FLINK-36812 [1], I've drafted a FLIP to address the > limitations of the current scan.partition.column implementation. > The existing approach splits data by equal numeric intervals, which breaks > down under two common scenarios: skewed data distribution and the absence > of a suitable numeric column (e.g., string-typed partition keys). > > Two extensions are proposed: > > *1. Physical ID Partitioning (scan.partition.use-physical-id)* > For databases that expose a physical row identifier (Oracle ROWID, > PostgreSQL ctid), the connector can derive partition bounds automatically > from DB-native metadata. > Since rows are physically distributed across storage blocks, this produces > naturally balanced splits with no user-defined partition column required. > > *2. Boundary Query (scan.partition.boundary-query)* > A SQL query returning ordered split points for the column specified in > scan.partition.column. The connector uses these values to generate range > predicates, enabling distribution-aware parallel reads for any column type > — including strings. > scan.partition.num is required and determines how many partitions are > created. > This is conceptually close to Apache Sqoop's --boundary-query, extended to > support string-type columns natively. > > Both extensions leave existing behavior unchanged. > > Full design: [2] > > Feedback and questions are welcome. > > Thanks, > Chanhae Oh > > [1] https://issues.apache.org/jira/browse/FLINK-36812 > [2] > > https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing >
