Hi João,
Thank you for sharing this — I really appreciated you reaching out. I went through the PR and I think the ConnectionProvider abstraction and the AsyncSnapshotSplitterEnumerator structure could serve as a solid reference point for what we're building, even though our FLIP targets the Table API layer (scan.partition.* options) rather than the DataStream API. One thing that stood out to me is how your implementation automates the PK-based lower/upper bound detection, which users currently have to provide manually. That fits well with the broader motivation of our FLIP — making equal-interval partitioning more accessible. Where our approaches differ is that your PR navigates data by depth (row count per chunk along the PK axis), while our FLIP focuses on distribution — letting users define split points based on where data is actually concentrated, or using physical block location (Oracle ROWID / PostgreSQL ctid) to partition without any partition column at all. I think the two approaches are complementary rather than overlapping, and combining the layers could give users a much richer set of options for different ingestion scenarios. Would love to keep the conversation going as both sides move forward. Best regards, ChanHae Oh > 2026. 7. 10. 오전 1:10, João Boto <[email protected]> 작성: > > Hi Chanhae > > Using the new SplitEnumerator, I have created this: > https://github.com/apache/flink-connector-jdbc/pull/209 > > I think that could feet on what you are trying to do.. > > As note, Im using it right now with good results.. > > Best > >> On 2026/07/08 11:12:07 Chanhae Oh wrote: >> Hi Leonard, >> >> Thank you for taking the time to review this proposal. >> Your feedback touched on the exact boundary conditions and correctness >> guarantees that I had not articulated precisely enough, and I genuinely >> learned from the breadth of your perspective. >> >> I have updated the FLIP to address each of your questions (changed sections >> are highlighted in yellow). >> >> 1. Physical ID Partitioning — correctness under concurrent changes >> (Updated: Proposed Changes > Physical ID Partitioning > Correctness >> Guarantees and Limitations) >> >> The two databases behave differently at the physical layer. Oracle ROWID is >> stable under normal DML — it only changes under structural operations >> (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid, however, >> changes on every UPDATE since a new tuple is written at a new location. >> Physical-id partitioning for PostgreSQL is therefore only safe when no >> concurrent DML runs during the scan. >> >> 2.1. Boundary-query — should the query return exactly N-1 values? >> (Updated: Proposed Changes > Boundary Query > Partition count behavior) >> >> Requiring exactly N-1 is too strict. Fewer values are allowed — the >> partition count simply reduces to (returned values) + 1. More than N-1 >> values is a validation error, since scan.partition.num serves as a resource >> cap and silently exceeding it could create more parallelism than the user >> intended. >> >> 2.2. Boundary-query — how are inclusive/exclusive predicates generated? >> (Updated: Proposed Changes > Boundary Query > Predicate contract) >> >> Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1) <= >> col < vi (left-inclusive, right-exclusive), and the last partition uses col >>> = v(N-1). This ensures no row is emitted by two partitions simultaneously. >> >> 2.3. Boundary-query — NULL values, duplicate boundaries, unsorted results, >> count mismatch >> (Updated: Proposed Changes > Boundary Query > Predicate contract, Partition >> count behavior) >> >> - NULL: Routed to Partition 1 via the explicit col IS NULL clause, since >> SQL NULL comparisons evaluate to NULL rather than TRUE. >> - Duplicate boundary values: Handled gracefully by the predicate pattern — >> the affected middle partition produces an empty result set with no data >> loss. >> - Unsorted results: The connector wraps the boundary-query in a subquery >> with ORDER BY 1, so the user does not need to include ORDER BY. >> - Count mismatch: Same as 2.1 — fewer values reduce partition count; more >> values are rejected at validation. >> >> Additionally, I added an Option Validation Matrix section to make the >> constraints across the three partitioning modes explicit in one place. >> >> Please let me know if any of the updated content raises further questions. >> >> Thank you again for your thorough review. >> >> P.S Thank you for your interest Youngil Kim. >> >> Best regards, >> ChanHae Oh >> >>> On Wed, Jul 8, 2026 at 2:48 PM Leonard Xu <[email protected]> wrote: >>> >>> Hi Chanhae, >>> >>> Thanks for driving this FLIP. The motivation makes sense to me: the current >>> equal-width JDBC partitioning does not work well for skewed data, and it >>> also >>> does not help much when users do not have a suitable numeric/date/timestamp >>> partition column. >>> >>> I am generally supportive of the direction, but I think both proposed >>> approaches >>> need clearer semantics. >>> >>> For physical-id partitioning, ROWID/ctid are physical identifiers rather >>> than >>> stable logical keys. They may change after updates, table rewrites, >>> vacuum, or >>> row movement. Since different splits may be read by different statements or >>> connections, the FLIP should clarify whether this mode only targets static >>> tables, or whether the connector can provide a consistent snapshot across >>> all >>> splits. Otherwise, missing or duplicated rows may be possible. >>> >>> For boundary-query partitioning, the contract should also be more precise. >>> For >>> example, if scan.partition.num is N, should the query return N - 1 >>> boundaries? >>> How are inclusive/exclusive predicates generated? How should NULL values, >>> duplicate boundaries, unordered results, or boundary count mismatches be >>> handled? >>> These details are important to avoid missing or duplicated rows. >>> >>> It would also be helpful to define the option validation matrix clearly, >>> e.g. >>> which options are required or mutually exclusive for range partitioning, >>> boundary-query partitioning, and physical-id partitioning. >>> >>> Overall, I like the direction, but I think the FLIP should first nail down >>> the >>> correctness guarantees and edge-case behavior. >>> >>> Best, >>> Leonard >>
