Hi Chanhae Using the new SplitEnumerator, I have created this: https://github.com/apache/flink-connector-jdbc/pull/209
I think that could feet on what you are trying to do.. As note, Im using it right now with good results.. Best On 2026/07/08 11:12:07 Chanhae Oh wrote: > Hi Leonard, > > Thank you for taking the time to review this proposal. > Your feedback touched on the exact boundary conditions and correctness > guarantees that I had not articulated precisely enough, and I genuinely > learned from the breadth of your perspective. > > I have updated the FLIP to address each of your questions (changed sections > are highlighted in yellow). > > 1. Physical ID Partitioning — correctness under concurrent changes > (Updated: Proposed Changes > Physical ID Partitioning > Correctness > Guarantees and Limitations) > > The two databases behave differently at the physical layer. Oracle ROWID is > stable under normal DML — it only changes under structural operations > (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid, however, > changes on every UPDATE since a new tuple is written at a new location. > Physical-id partitioning for PostgreSQL is therefore only safe when no > concurrent DML runs during the scan. > > 2.1. Boundary-query — should the query return exactly N-1 values? > (Updated: Proposed Changes > Boundary Query > Partition count behavior) > > Requiring exactly N-1 is too strict. Fewer values are allowed — the > partition count simply reduces to (returned values) + 1. More than N-1 > values is a validation error, since scan.partition.num serves as a resource > cap and silently exceeding it could create more parallelism than the user > intended. > > 2.2. Boundary-query — how are inclusive/exclusive predicates generated? > (Updated: Proposed Changes > Boundary Query > Predicate contract) > > Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1) <= > col < vi (left-inclusive, right-exclusive), and the last partition uses col > >= v(N-1). This ensures no row is emitted by two partitions simultaneously. > > 2.3. Boundary-query — NULL values, duplicate boundaries, unsorted results, > count mismatch > (Updated: Proposed Changes > Boundary Query > Predicate contract, Partition > count behavior) > > - NULL: Routed to Partition 1 via the explicit col IS NULL clause, since > SQL NULL comparisons evaluate to NULL rather than TRUE. > - Duplicate boundary values: Handled gracefully by the predicate pattern — > the affected middle partition produces an empty result set with no data > loss. > - Unsorted results: The connector wraps the boundary-query in a subquery > with ORDER BY 1, so the user does not need to include ORDER BY. > - Count mismatch: Same as 2.1 — fewer values reduce partition count; more > values are rejected at validation. > > Additionally, I added an Option Validation Matrix section to make the > constraints across the three partitioning modes explicit in one place. > > Please let me know if any of the updated content raises further questions. > > Thank you again for your thorough review. > > P.S Thank you for your interest Youngil Kim. > > Best regards, > ChanHae Oh > > On Wed, Jul 8, 2026 at 2:48 PM Leonard Xu <[email protected]> wrote: > > > Hi Chanhae, > > > > Thanks for driving this FLIP. The motivation makes sense to me: the current > > equal-width JDBC partitioning does not work well for skewed data, and it > > also > > does not help much when users do not have a suitable numeric/date/timestamp > > partition column. > > > > I am generally supportive of the direction, but I think both proposed > > approaches > > need clearer semantics. > > > > For physical-id partitioning, ROWID/ctid are physical identifiers rather > > than > > stable logical keys. They may change after updates, table rewrites, > > vacuum, or > > row movement. Since different splits may be read by different statements or > > connections, the FLIP should clarify whether this mode only targets static > > tables, or whether the connector can provide a consistent snapshot across > > all > > splits. Otherwise, missing or duplicated rows may be possible. > > > > For boundary-query partitioning, the contract should also be more precise. > > For > > example, if scan.partition.num is N, should the query return N - 1 > > boundaries? > > How are inclusive/exclusive predicates generated? How should NULL values, > > duplicate boundaries, unordered results, or boundary count mismatches be > > handled? > > These details are important to avoid missing or duplicated rows. > > > > It would also be helpful to define the option validation matrix clearly, > > e.g. > > which options are required or mutually exclusive for range partitioning, > > boundary-query partitioning, and physical-id partitioning. > > > > Overall, I like the direction, but I think the FLIP should first nail down > > the > > correctness guarantees and edge-case behavior. > > > > Best, > > Leonard >
