Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Chanhae Oh]

Hi Leonard,


Thank you for confirming — and for facilitating the Confluence access so
smoothly.

The FLIP[1] wiki page is now in place (Yuepeng Pan helped set it up, and I
have completed the edits).

I will be moving forward with the vote shortly.


[1] cwiki.apache.org/confluence/x/x4Q_Gg


Best regards,

ChanHae Oh

2026년 7월 22일 수요일, Leonard Xu 님이 작성:

>
> > I have already signed up on Apache Flink Confluence [1] under the
> > username Chanhae
> > Oh.
> >
> > If you are able to grant me write access, I will move the current draft
> > there using the FLIP template and proceed toward a community vote.
>
> Done, you can have a try.
>
> Best,
> Leonard
>
>
> >
> > Best regards,
> >
> > Chanhae
> >
> > On Tue, Jul 21, 2026 at 12:16 PM Leonard Xu  wrote:
> >
> >> Hi ChanHae,
> >>
> >> Thanks for the update. The newly added limitations and boundary-query
> >> result
> >> requirements address my remaining concerns. No further concerns from me.
> >>
> >> One small procedural suggestion: FLIPs are usually maintained in the
> Apache
> >> Flink Confluence wiki, so it would be great if you could move the
> current
> >> Google Docs draft there using the FLIP template [1]. This will make it
> >> easier
> >> for the community to review, track, and eventually vote on the FLIP.
> >>
> >> If you do not have write permission to the Flink wiki yet, no worries.
> >> Please
> >> feel free to share your Apache Confluence username, and I can help add
> the
> >> permission for you.
> >>
> >> Best,
> >> Leonard
> >>
> >> [1]
> >> https://cwiki.apache.org/confluence/spaces/FLINK/pages/
> 65145553/FLIP+Template
> >>
> >>> 2026 7月 20 11:41 下午，Chanhae Oh  写道：
> >>>
> >>> Hi Leonard,
> >>>
> >>> Thank you for the +1 and for the additional suggestions.
> >>>
> >>> I have updated the FLIP based on your comments (changes marked with
> green
> >>> underlines):
> >>>
> >>> - Physical-id partitioning: added explicit limitation note for
> >>> static/write-free tables
> >>> - Boundary-query partitioning: added result requirements (single
> column,
> >>> type compatibility, NULL rejection)
> >>>
> >>> Please let me know if there is anything further needed before moving to
> >> the
> >>> next stage.
> >>>
> >>> Best regards,
> >>> Chanhae Oh
> >>>
> >>> On Mon, Jul 20, 2026 at 1:19 PM Leonard Xu  wrote:
> >>>
>  Hi ChanHae,
> 
>  Thanks for the update. With these semantics clarified, I think the
>  proposal is
>  reasonable to move forward.
> 
>  My remaining suggestion is to make the limitations very explicit in
> both
>  the
>  FLIP and the user documentation. In particular, physical-id
> partitioning
>  should
>  be documented as suitable for static tables, or tables without
> >> concurrent
>  changes during the scan. For boundary-query partitioning, the
> connector
>  should
>  reject NULL boundary values and require the boundary query to return a
>  single
>  column that is comparable with the partition column.
> 
>  With these documented, I am +1 on moving this FLIP forward.
> 
>  Best,
>  Leonard
> 
> > 2026 7月 8 7:12 下午，Chanhae Oh  写道：
> >
> > Hi Leonard,
> >
> > Thank you for taking the time to review this proposal.
> > Your feedback touched on the exact boundary conditions and
> correctness
> > guarantees that I had not articulated precisely enough, and I
> genuinely
> > learned from the breadth of your perspective.
> >
> > I have updated the FLIP to address each of your questions (changed
>  sections
> > are highlighted in yellow).
> >
> > 1. Physical ID Partitioning — correctness under concurrent changes
> > (Updated: Proposed Changes > Physical ID Partitioning > Correctness
> > Guarantees and Limitations)
> >
> > The two databases behave differently at the physical layer. Oracle
> >> ROWID
>  is
> > stable under normal DML — it only changes under structural operations
> > (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid,
>  however,
> > changes on every UPDATE since a new tuple is written at a new
> location.
> > Physical-id partitioning for PostgreSQL is therefore only safe when
> no
> > concurrent DML runs during the scan.
> >
> > 2.1. Boundary-query — should the query return exactly N-1 values?
> > (Updated: Proposed Changes > Boundary Query > Partition count
> behavior)
> >
> > Requiring exactly N-1 is too strict. Fewer values are allowed — the
> > partition count simply reduces to (returned values) + 1. More than
> N-1
> > values is a validation error, since scan.partition.num serves as a
>  resource
> > cap and silently exceeding it could create more parallelism than the
> >> user
> > intended.
> >
> > 2.2. Boundary-query — how are inclusive/exclusive predicates
> generated?
> > (Updated: Proposed Changes > Boundary Query > Predicate contract)
> >
> > Partition 1 use

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Leonard Xu]

> I have already signed up on Apache Flink Confluence [1] under the
> username Chanhae
> Oh.
> 
> If you are able to grant me write access, I will move the current draft
> there using the FLIP template and proceed toward a community vote.

Done, you can have a try.

Best,
Leonard


> 
> Best regards,
> 
> Chanhae
> 
> On Tue, Jul 21, 2026 at 12:16 PM Leonard Xu  wrote:
> 
>> Hi ChanHae,
>> 
>> Thanks for the update. The newly added limitations and boundary-query
>> result
>> requirements address my remaining concerns. No further concerns from me.
>> 
>> One small procedural suggestion: FLIPs are usually maintained in the Apache
>> Flink Confluence wiki, so it would be great if you could move the current
>> Google Docs draft there using the FLIP template [1]. This will make it
>> easier
>> for the community to review, track, and eventually vote on the FLIP.
>> 
>> If you do not have write permission to the Flink wiki yet, no worries.
>> Please
>> feel free to share your Apache Confluence username, and I can help add the
>> permission for you.
>> 
>> Best,
>> Leonard
>> 
>> [1]
>> https://cwiki.apache.org/confluence/spaces/FLINK/pages/65145553/FLIP+Template
>> 
>>> 2026 7月 20 11:41 下午，Chanhae Oh  写道：
>>> 
>>> Hi Leonard,
>>> 
>>> Thank you for the +1 and for the additional suggestions.
>>> 
>>> I have updated the FLIP based on your comments (changes marked with green
>>> underlines):
>>> 
>>> - Physical-id partitioning: added explicit limitation note for
>>> static/write-free tables
>>> - Boundary-query partitioning: added result requirements (single column,
>>> type compatibility, NULL rejection)
>>> 
>>> Please let me know if there is anything further needed before moving to
>> the
>>> next stage.
>>> 
>>> Best regards,
>>> Chanhae Oh
>>> 
>>> On Mon, Jul 20, 2026 at 1:19 PM Leonard Xu  wrote:
>>> 
 Hi ChanHae,
 
 Thanks for the update. With these semantics clarified, I think the
 proposal is
 reasonable to move forward.
 
 My remaining suggestion is to make the limitations very explicit in both
 the
 FLIP and the user documentation. In particular, physical-id partitioning
 should
 be documented as suitable for static tables, or tables without
>> concurrent
 changes during the scan. For boundary-query partitioning, the connector
 should
 reject NULL boundary values and require the boundary query to return a
 single
 column that is comparable with the partition column.
 
 With these documented, I am +1 on moving this FLIP forward.
 
 Best,
 Leonard
 
> 2026 7月 8 7:12 下午，Chanhae Oh  写道：
> 
> Hi Leonard,
> 
> Thank you for taking the time to review this proposal.
> Your feedback touched on the exact boundary conditions and correctness
> guarantees that I had not articulated precisely enough, and I genuinely
> learned from the breadth of your perspective.
> 
> I have updated the FLIP to address each of your questions (changed
 sections
> are highlighted in yellow).
> 
> 1. Physical ID Partitioning — correctness under concurrent changes
> (Updated: Proposed Changes > Physical ID Partitioning > Correctness
> Guarantees and Limitations)
> 
> The two databases behave differently at the physical layer. Oracle
>> ROWID
 is
> stable under normal DML — it only changes under structural operations
> (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid,
 however,
> changes on every UPDATE since a new tuple is written at a new location.
> Physical-id partitioning for PostgreSQL is therefore only safe when no
> concurrent DML runs during the scan.
> 
> 2.1. Boundary-query — should the query return exactly N-1 values?
> (Updated: Proposed Changes > Boundary Query > Partition count behavior)
> 
> Requiring exactly N-1 is too strict. Fewer values are allowed — the
> partition count simply reduces to (returned values) + 1. More than N-1
> values is a validation error, since scan.partition.num serves as a
 resource
> cap and silently exceeding it could create more parallelism than the
>> user
> intended.
> 
> 2.2. Boundary-query — how are inclusive/exclusive predicates generated?
> (Updated: Proposed Changes > Boundary Query > Predicate contract)
> 
> Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1)
>> <=
> col < vi (left-inclusive, right-exclusive), and the last partition uses
 col
>> = v(N-1). This ensures no row is emitted by two partitions
 simultaneously.
> 
> 2.3. Boundary-query — NULL values, duplicate boundaries, unsorted
 results,
> count mismatch
> (Updated: Proposed Changes > Boundary Query > Predicate contract,
 Partition
> count behavior)
> 
> - NULL: Routed to Partition 1 via the explicit col IS NULL clause,
>> since
> SQL NULL comparisons evaluate to NULL rather than TRUE.
> 

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Chanhae Oh]

Hi Leonard,


Thank you so much for all your support and guidance.


I have already signed up on Apache Flink Confluence [1] under the
username Chanhae
Oh.

If you are able to grant me write access, I will move the current draft
there using the FLIP template and proceed toward a community vote.


[1]
 
https://cwiki.apache.org/confluence/spaces/FLINK/pages/44302796/Apache+Flink+Home



Thank you again for your kind help!


Best regards,

Chanhae

On Tue, Jul 21, 2026 at 12:16 PM Leonard Xu  wrote:

> Hi ChanHae,
>
> Thanks for the update. The newly added limitations and boundary-query
> result
> requirements address my remaining concerns. No further concerns from me.
>
> One small procedural suggestion: FLIPs are usually maintained in the Apache
> Flink Confluence wiki, so it would be great if you could move the current
> Google Docs draft there using the FLIP template [1]. This will make it
> easier
> for the community to review, track, and eventually vote on the FLIP.
>
> If you do not have write permission to the Flink wiki yet, no worries.
> Please
> feel free to share your Apache Confluence username, and I can help add the
> permission for you.
>
> Best,
> Leonard
>
> [1]
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/65145553/FLIP+Template
>
> > 2026 7月 20 11:41 下午，Chanhae Oh  写道：
> >
> > Hi Leonard,
> >
> > Thank you for the +1 and for the additional suggestions.
> >
> > I have updated the FLIP based on your comments (changes marked with green
> > underlines):
> >
> > - Physical-id partitioning: added explicit limitation note for
> > static/write-free tables
> > - Boundary-query partitioning: added result requirements (single column,
> > type compatibility, NULL rejection)
> >
> > Please let me know if there is anything further needed before moving to
> the
> > next stage.
> >
> > Best regards,
> > Chanhae Oh
> >
> > On Mon, Jul 20, 2026 at 1:19 PM Leonard Xu  wrote:
> >
> >> Hi ChanHae,
> >>
> >> Thanks for the update. With these semantics clarified, I think the
> >> proposal is
> >> reasonable to move forward.
> >>
> >> My remaining suggestion is to make the limitations very explicit in both
> >> the
> >> FLIP and the user documentation. In particular, physical-id partitioning
> >> should
> >> be documented as suitable for static tables, or tables without
> concurrent
> >> changes during the scan. For boundary-query partitioning, the connector
> >> should
> >> reject NULL boundary values and require the boundary query to return a
> >> single
> >> column that is comparable with the partition column.
> >>
> >> With these documented, I am +1 on moving this FLIP forward.
> >>
> >> Best,
> >> Leonard
> >>
> >>> 2026 7月 8 7:12 下午，Chanhae Oh  写道：
> >>>
> >>> Hi Leonard,
> >>>
> >>> Thank you for taking the time to review this proposal.
> >>> Your feedback touched on the exact boundary conditions and correctness
> >>> guarantees that I had not articulated precisely enough, and I genuinely
> >>> learned from the breadth of your perspective.
> >>>
> >>> I have updated the FLIP to address each of your questions (changed
> >> sections
> >>> are highlighted in yellow).
> >>>
> >>> 1. Physical ID Partitioning — correctness under concurrent changes
> >>> (Updated: Proposed Changes > Physical ID Partitioning > Correctness
> >>> Guarantees and Limitations)
> >>>
> >>> The two databases behave differently at the physical layer. Oracle
> ROWID
> >> is
> >>> stable under normal DML — it only changes under structural operations
> >>> (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid,
> >> however,
> >>> changes on every UPDATE since a new tuple is written at a new location.
> >>> Physical-id partitioning for PostgreSQL is therefore only safe when no
> >>> concurrent DML runs during the scan.
> >>>
> >>> 2.1. Boundary-query — should the query return exactly N-1 values?
> >>> (Updated: Proposed Changes > Boundary Query > Partition count behavior)
> >>>
> >>> Requiring exactly N-1 is too strict. Fewer values are allowed — the
> >>> partition count simply reduces to (returned values) + 1. More than N-1
> >>> values is a validation error, since scan.partition.num serves as a
> >> resource
> >>> cap and silently exceeding it could create more parallelism than the
> user
> >>> intended.
> >>>
> >>> 2.2. Boundary-query — how are inclusive/exclusive predicates generated?
> >>> (Updated: Proposed Changes > Boundary Query > Predicate contract)
> >>>
> >>> Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1)
> <=
> >>> col < vi (left-inclusive, right-exclusive), and the last partition uses
> >> col
>  = v(N-1). This ensures no row is emitted by two partitions
> >> simultaneously.
> >>>
> >>> 2.3. Boundary-query — NULL values, duplicate boundaries, unsorted
> >> results,
> >>> count mismatch
> >>> (Updated: Proposed Changes > Boundary Query > Predicate contract,
> >> Partition
> >>> count behavior)
> >>

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Leonard Xu]

Hi ChanHae,

Thanks for the update. The newly added limitations and boundary-query result
requirements address my remaining concerns. No further concerns from me.

One small procedural suggestion: FLIPs are usually maintained in the Apache
Flink Confluence wiki, so it would be great if you could move the current
Google Docs draft there using the FLIP template [1]. This will make it easier
for the community to review, track, and eventually vote on the FLIP.

If you do not have write permission to the Flink wiki yet, no worries. Please
feel free to share your Apache Confluence username, and I can help add the
permission for you.

Best,
Leonard

[1] 
https://cwiki.apache.org/confluence/spaces/FLINK/pages/65145553/FLIP+Template

> 2026 7月 20 11:41 下午，Chanhae Oh  写道：
> 
> Hi Leonard,
> 
> Thank you for the +1 and for the additional suggestions.
> 
> I have updated the FLIP based on your comments (changes marked with green
> underlines):
> 
> - Physical-id partitioning: added explicit limitation note for
> static/write-free tables
> - Boundary-query partitioning: added result requirements (single column,
> type compatibility, NULL rejection)
> 
> Please let me know if there is anything further needed before moving to the
> next stage.
> 
> Best regards,
> Chanhae Oh
> 
> On Mon, Jul 20, 2026 at 1:19 PM Leonard Xu  wrote:
> 
>> Hi ChanHae,
>> 
>> Thanks for the update. With these semantics clarified, I think the
>> proposal is
>> reasonable to move forward.
>> 
>> My remaining suggestion is to make the limitations very explicit in both
>> the
>> FLIP and the user documentation. In particular, physical-id partitioning
>> should
>> be documented as suitable for static tables, or tables without concurrent
>> changes during the scan. For boundary-query partitioning, the connector
>> should
>> reject NULL boundary values and require the boundary query to return a
>> single
>> column that is comparable with the partition column.
>> 
>> With these documented, I am +1 on moving this FLIP forward.
>> 
>> Best,
>> Leonard
>> 
>>> 2026 7月 8 7:12 下午，Chanhae Oh  写道：
>>> 
>>> Hi Leonard,
>>> 
>>> Thank you for taking the time to review this proposal.
>>> Your feedback touched on the exact boundary conditions and correctness
>>> guarantees that I had not articulated precisely enough, and I genuinely
>>> learned from the breadth of your perspective.
>>> 
>>> I have updated the FLIP to address each of your questions (changed
>> sections
>>> are highlighted in yellow).
>>> 
>>> 1. Physical ID Partitioning — correctness under concurrent changes
>>> (Updated: Proposed Changes > Physical ID Partitioning > Correctness
>>> Guarantees and Limitations)
>>> 
>>> The two databases behave differently at the physical layer. Oracle ROWID
>> is
>>> stable under normal DML — it only changes under structural operations
>>> (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid,
>> however,
>>> changes on every UPDATE since a new tuple is written at a new location.
>>> Physical-id partitioning for PostgreSQL is therefore only safe when no
>>> concurrent DML runs during the scan.
>>> 
>>> 2.1. Boundary-query — should the query return exactly N-1 values?
>>> (Updated: Proposed Changes > Boundary Query > Partition count behavior)
>>> 
>>> Requiring exactly N-1 is too strict. Fewer values are allowed — the
>>> partition count simply reduces to (returned values) + 1. More than N-1
>>> values is a validation error, since scan.partition.num serves as a
>> resource
>>> cap and silently exceeding it could create more parallelism than the user
>>> intended.
>>> 
>>> 2.2. Boundary-query — how are inclusive/exclusive predicates generated?
>>> (Updated: Proposed Changes > Boundary Query > Predicate contract)
>>> 
>>> Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1) <=
>>> col < vi (left-inclusive, right-exclusive), and the last partition uses
>> col
 = v(N-1). This ensures no row is emitted by two partitions
>> simultaneously.
>>> 
>>> 2.3. Boundary-query — NULL values, duplicate boundaries, unsorted
>> results,
>>> count mismatch
>>> (Updated: Proposed Changes > Boundary Query > Predicate contract,
>> Partition
>>> count behavior)
>>> 
>>> - NULL: Routed to Partition 1 via the explicit col IS NULL clause, since
>>> SQL NULL comparisons evaluate to NULL rather than TRUE.
>>> - Duplicate boundary values: Handled gracefully by the predicate pattern
>> —
>>> the affected middle partition produces an empty result set with no data
>>> loss.
>>> - Unsorted results: The connector wraps the boundary-query in a subquery
>>> with ORDER BY 1, so the user does not need to include ORDER BY.
>>> - Count mismatch: Same as 2.1 — fewer values reduce partition count; more
>>> values are rejected at validation.
>>> 
>>> Additionally, I added an Option Validation Matrix section to make the
>>> constraints across the three partitioning modes explicit in one place.
>>> 
>>> Please let me know if any of the updated content raises further
>> quest

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Chanhae Oh]

Hi Leonard,

Thank you for the +1 and for the additional suggestions.

I have updated the FLIP based on your comments (changes marked with green
underlines):

- Physical-id partitioning: added explicit limitation note for
static/write-free tables
- Boundary-query partitioning: added result requirements (single column,
type compatibility, NULL rejection)

Please let me know if there is anything further needed before moving to the
next stage.

Best regards,
Chanhae Oh

On Mon, Jul 20, 2026 at 1:19 PM Leonard Xu  wrote:

> Hi ChanHae,
>
> Thanks for the update. With these semantics clarified, I think the
> proposal is
> reasonable to move forward.
>
> My remaining suggestion is to make the limitations very explicit in both
> the
> FLIP and the user documentation. In particular, physical-id partitioning
> should
> be documented as suitable for static tables, or tables without concurrent
> changes during the scan. For boundary-query partitioning, the connector
> should
> reject NULL boundary values and require the boundary query to return a
> single
> column that is comparable with the partition column.
>
> With these documented, I am +1 on moving this FLIP forward.
>
> Best,
> Leonard
>
> > 2026 7月 8 7:12 下午，Chanhae Oh  写道：
> >
> > Hi Leonard,
> >
> > Thank you for taking the time to review this proposal.
> > Your feedback touched on the exact boundary conditions and correctness
> > guarantees that I had not articulated precisely enough, and I genuinely
> > learned from the breadth of your perspective.
> >
> > I have updated the FLIP to address each of your questions (changed
> sections
> > are highlighted in yellow).
> >
> > 1. Physical ID Partitioning — correctness under concurrent changes
> > (Updated: Proposed Changes > Physical ID Partitioning > Correctness
> > Guarantees and Limitations)
> >
> > The two databases behave differently at the physical layer. Oracle ROWID
> is
> > stable under normal DML — it only changes under structural operations
> > (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid,
> however,
> > changes on every UPDATE since a new tuple is written at a new location.
> > Physical-id partitioning for PostgreSQL is therefore only safe when no
> > concurrent DML runs during the scan.
> >
> > 2.1. Boundary-query — should the query return exactly N-1 values?
> > (Updated: Proposed Changes > Boundary Query > Partition count behavior)
> >
> > Requiring exactly N-1 is too strict. Fewer values are allowed — the
> > partition count simply reduces to (returned values) + 1. More than N-1
> > values is a validation error, since scan.partition.num serves as a
> resource
> > cap and silently exceeding it could create more parallelism than the user
> > intended.
> >
> > 2.2. Boundary-query — how are inclusive/exclusive predicates generated?
> > (Updated: Proposed Changes > Boundary Query > Predicate contract)
> >
> > Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1) <=
> > col < vi (left-inclusive, right-exclusive), and the last partition uses
> col
> >> = v(N-1). This ensures no row is emitted by two partitions
> simultaneously.
> >
> > 2.3. Boundary-query — NULL values, duplicate boundaries, unsorted
> results,
> > count mismatch
> > (Updated: Proposed Changes > Boundary Query > Predicate contract,
> Partition
> > count behavior)
> >
> > - NULL: Routed to Partition 1 via the explicit col IS NULL clause, since
> > SQL NULL comparisons evaluate to NULL rather than TRUE.
> > - Duplicate boundary values: Handled gracefully by the predicate pattern
> —
> > the affected middle partition produces an empty result set with no data
> > loss.
> > - Unsorted results: The connector wraps the boundary-query in a subquery
> > with ORDER BY 1, so the user does not need to include ORDER BY.
> > - Count mismatch: Same as 2.1 — fewer values reduce partition count; more
> > values are rejected at validation.
> >
> > Additionally, I added an Option Validation Matrix section to make the
> > constraints across the three partitioning modes explicit in one place.
> >
> > Please let me know if any of the updated content raises further
> questions.
> >
> > Thank you again for your thorough review.
> >
> > P.S Thank you for your interest Youngil Kim.
> >
> > Best regards,
> > ChanHae Oh
> >
> > On Wed, Jul 8, 2026 at 2:48 PM Leonard Xu  wrote:
> >
> >> Hi Chanhae,
> >>
> >> Thanks for driving this FLIP. The motivation makes sense to me: the
> current
> >> equal-width JDBC partitioning does not work well for skewed data, and it
> >> also
> >> does not help much when users do not have a suitable
> numeric/date/timestamp
> >> partition column.
> >>
> >> I am generally supportive of the direction, but I think both proposed
> >> approaches
> >> need clearer semantics.
> >>
> >> For physical-id partitioning, ROWID/ctid are physical identifiers rather
> >> than
> >> stable logical keys. They may change after updates, table rewrites,
> >> vacuum, or
> >> row movement. Since different splits may be rea

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Leonard Xu]

Hi ChanHae,

Thanks for the update. With these semantics clarified, I think the proposal is
reasonable to move forward.

My remaining suggestion is to make the limitations very explicit in both the
FLIP and the user documentation. In particular, physical-id partitioning should
be documented as suitable for static tables, or tables without concurrent
changes during the scan. For boundary-query partitioning, the connector should
reject NULL boundary values and require the boundary query to return a single
column that is comparable with the partition column.

With these documented, I am +1 on moving this FLIP forward.

Best,
Leonard

> 2026 7月 8 7:12 下午，Chanhae Oh  写道：
> 
> Hi Leonard,
> 
> Thank you for taking the time to review this proposal.
> Your feedback touched on the exact boundary conditions and correctness
> guarantees that I had not articulated precisely enough, and I genuinely
> learned from the breadth of your perspective.
> 
> I have updated the FLIP to address each of your questions (changed sections
> are highlighted in yellow).
> 
> 1. Physical ID Partitioning — correctness under concurrent changes
> (Updated: Proposed Changes > Physical ID Partitioning > Correctness
> Guarantees and Limitations)
> 
> The two databases behave differently at the physical layer. Oracle ROWID is
> stable under normal DML — it only changes under structural operations
> (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid, however,
> changes on every UPDATE since a new tuple is written at a new location.
> Physical-id partitioning for PostgreSQL is therefore only safe when no
> concurrent DML runs during the scan.
> 
> 2.1. Boundary-query — should the query return exactly N-1 values?
> (Updated: Proposed Changes > Boundary Query > Partition count behavior)
> 
> Requiring exactly N-1 is too strict. Fewer values are allowed — the
> partition count simply reduces to (returned values) + 1. More than N-1
> values is a validation error, since scan.partition.num serves as a resource
> cap and silently exceeding it could create more parallelism than the user
> intended.
> 
> 2.2. Boundary-query — how are inclusive/exclusive predicates generated?
> (Updated: Proposed Changes > Boundary Query > Predicate contract)
> 
> Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1) <=
> col < vi (left-inclusive, right-exclusive), and the last partition uses col
>> = v(N-1). This ensures no row is emitted by two partitions simultaneously.
> 
> 2.3. Boundary-query — NULL values, duplicate boundaries, unsorted results,
> count mismatch
> (Updated: Proposed Changes > Boundary Query > Predicate contract, Partition
> count behavior)
> 
> - NULL: Routed to Partition 1 via the explicit col IS NULL clause, since
> SQL NULL comparisons evaluate to NULL rather than TRUE.
> - Duplicate boundary values: Handled gracefully by the predicate pattern —
> the affected middle partition produces an empty result set with no data
> loss.
> - Unsorted results: The connector wraps the boundary-query in a subquery
> with ORDER BY 1, so the user does not need to include ORDER BY.
> - Count mismatch: Same as 2.1 — fewer values reduce partition count; more
> values are rejected at validation.
> 
> Additionally, I added an Option Validation Matrix section to make the
> constraints across the three partitioning modes explicit in one place.
> 
> Please let me know if any of the updated content raises further questions.
> 
> Thank you again for your thorough review.
> 
> P.S Thank you for your interest Youngil Kim.
> 
> Best regards,
> ChanHae Oh
> 
> On Wed, Jul 8, 2026 at 2:48 PM Leonard Xu  wrote:
> 
>> Hi Chanhae,
>> 
>> Thanks for driving this FLIP. The motivation makes sense to me: the current
>> equal-width JDBC partitioning does not work well for skewed data, and it
>> also
>> does not help much when users do not have a suitable numeric/date/timestamp
>> partition column.
>> 
>> I am generally supportive of the direction, but I think both proposed
>> approaches
>> need clearer semantics.
>> 
>> For physical-id partitioning, ROWID/ctid are physical identifiers rather
>> than
>> stable logical keys. They may change after updates, table rewrites,
>> vacuum, or
>> row movement. Since different splits may be read by different statements or
>> connections, the FLIP should clarify whether this mode only targets static
>> tables, or whether the connector can provide a consistent snapshot across
>> all
>> splits. Otherwise, missing or duplicated rows may be possible.
>> 
>> For boundary-query partitioning, the contract should also be more precise.
>> For
>> example, if scan.partition.num is N, should the query return N - 1
>> boundaries?
>> How are inclusive/exclusive predicates generated? How should NULL values,
>> duplicate boundaries, unordered results, or boundary count mismatches be
>> handled?
>> These details are important to avoid missing or duplicated rows.
>> 
>> It would also be helpful to define the option validation matrix c

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Chanhae Oh]

Hi João,

Thank you for sharing this — I really appreciated you reaching out.

I went through the PR and I think the ConnectionProvider abstraction and the 
AsyncSnapshotSplitterEnumerator structure could serve as a solid reference 
point for what we're building, even though our FLIP targets the Table API layer 
(scan.partition.* options) rather than the DataStream API.

One thing that stood out to me is how your implementation automates the 
PK-based lower/upper bound detection, which users currently have to provide 
manually. That fits well with the broader motivation of our FLIP — making 
equal-interval partitioning more accessible.

Where our approaches differ is that your PR navigates data by depth (row count 
per chunk along the PK axis), while our FLIP focuses on distribution — letting 
users define split points based on where data is actually concentrated, or 
using physical block location (Oracle ROWID / PostgreSQL ctid) to partition 
without any partition column at all. I think the two approaches are 
complementary rather than overlapping, and combining the layers could give 
users a much richer set of options for different ingestion scenarios.

Would love to keep the conversation going as both sides move forward.

Best regards,
ChanHae Oh

> 2026. 7. 10. 오전 1:10, João Boto  작성:
> 
> Hi Chanhae
> 
> Using the new SplitEnumerator, I have created this:
> https://github.com/apache/flink-connector-jdbc/pull/209
> 
> I think that could feet on what you are trying to do..
> 
> As note, Im using it right now with good results..
> 
> Best
> 
>> On 2026/07/08 11:12:07 Chanhae Oh wrote:
>> Hi Leonard,
>> 
>> Thank you for taking the time to review this proposal.
>> Your feedback touched on the exact boundary conditions and correctness
>> guarantees that I had not articulated precisely enough, and I genuinely
>> learned from the breadth of your perspective.
>> 
>> I have updated the FLIP to address each of your questions (changed sections
>> are highlighted in yellow).
>> 
>> 1. Physical ID Partitioning — correctness under concurrent changes
>> (Updated: Proposed Changes > Physical ID Partitioning > Correctness
>> Guarantees and Limitations)
>> 
>> The two databases behave differently at the physical layer. Oracle ROWID is
>> stable under normal DML — it only changes under structural operations
>> (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid, however,
>> changes on every UPDATE since a new tuple is written at a new location.
>> Physical-id partitioning for PostgreSQL is therefore only safe when no
>> concurrent DML runs during the scan.
>> 
>> 2.1. Boundary-query — should the query return exactly N-1 values?
>> (Updated: Proposed Changes > Boundary Query > Partition count behavior)
>> 
>> Requiring exactly N-1 is too strict. Fewer values are allowed — the
>> partition count simply reduces to (returned values) + 1. More than N-1
>> values is a validation error, since scan.partition.num serves as a resource
>> cap and silently exceeding it could create more parallelism than the user
>> intended.
>> 
>> 2.2. Boundary-query — how are inclusive/exclusive predicates generated?
>> (Updated: Proposed Changes > Boundary Query > Predicate contract)
>> 
>> Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1) <=
>> col < vi (left-inclusive, right-exclusive), and the last partition uses col
>>> = v(N-1). This ensures no row is emitted by two partitions simultaneously.
>> 
>> 2.3. Boundary-query — NULL values, duplicate boundaries, unsorted results,
>> count mismatch
>> (Updated: Proposed Changes > Boundary Query > Predicate contract, Partition
>> count behavior)
>> 
>> - NULL: Routed to Partition 1 via the explicit col IS NULL clause, since
>> SQL NULL comparisons evaluate to NULL rather than TRUE.
>> - Duplicate boundary values: Handled gracefully by the predicate pattern —
>> the affected middle partition produces an empty result set with no data
>> loss.
>> - Unsorted results: The connector wraps the boundary-query in a subquery
>> with ORDER BY 1, so the user does not need to include ORDER BY.
>> - Count mismatch: Same as 2.1 — fewer values reduce partition count; more
>> values are rejected at validation.
>> 
>> Additionally, I added an Option Validation Matrix section to make the
>> constraints across the three partitioning modes explicit in one place.
>> 
>> Please let me know if any of the updated content raises further questions.
>> 
>> Thank you again for your thorough review.
>> 
>> P.S Thank you for your interest Youngil Kim.
>> 
>> Best regards,
>> ChanHae Oh
>> 
>>> On Wed, Jul 8, 2026 at 2:48 PM Leonard Xu  wrote:
>>> 
>>> Hi Chanhae,
>>> 
>>> Thanks for driving this FLIP. The motivation makes sense to me: the current
>>> equal-width JDBC partitioning does not work well for skewed data, and it
>>> also
>>> does not help much when users do not have a suitable numeric/date/timestamp
>>> partition column.
>>> 
>>> I am generally supportive of the direction, but I thin

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[João Boto]

Hi Chanhae

Using the new SplitEnumerator, I have created this:
https://github.com/apache/flink-connector-jdbc/pull/209

I think that could feet on what you are trying to do..

As note, Im using it right now with good results..

Best

On 2026/07/08 11:12:07 Chanhae Oh wrote:
> Hi Leonard,
> 
> Thank you for taking the time to review this proposal.
> Your feedback touched on the exact boundary conditions and correctness
> guarantees that I had not articulated precisely enough, and I genuinely
> learned from the breadth of your perspective.
> 
> I have updated the FLIP to address each of your questions (changed sections
> are highlighted in yellow).
> 
> 1. Physical ID Partitioning — correctness under concurrent changes
> (Updated: Proposed Changes > Physical ID Partitioning > Correctness
> Guarantees and Limitations)
> 
> The two databases behave differently at the physical layer. Oracle ROWID is
> stable under normal DML — it only changes under structural operations
> (ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid, however,
> changes on every UPDATE since a new tuple is written at a new location.
> Physical-id partitioning for PostgreSQL is therefore only safe when no
> concurrent DML runs during the scan.
> 
> 2.1. Boundary-query — should the query return exactly N-1 values?
> (Updated: Proposed Changes > Boundary Query > Partition count behavior)
> 
> Requiring exactly N-1 is too strict. Fewer values are allowed — the
> partition count simply reduces to (returned values) + 1. More than N-1
> values is a validation error, since scan.partition.num serves as a resource
> cap and silently exceeding it could create more parallelism than the user
> intended.
> 
> 2.2. Boundary-query — how are inclusive/exclusive predicates generated?
> (Updated: Proposed Changes > Boundary Query > Predicate contract)
> 
> Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1) <=
> col < vi (left-inclusive, right-exclusive), and the last partition uses col
> >= v(N-1). This ensures no row is emitted by two partitions simultaneously.
> 
> 2.3. Boundary-query — NULL values, duplicate boundaries, unsorted results,
> count mismatch
> (Updated: Proposed Changes > Boundary Query > Predicate contract, Partition
> count behavior)
> 
> - NULL: Routed to Partition 1 via the explicit col IS NULL clause, since
> SQL NULL comparisons evaluate to NULL rather than TRUE.
> - Duplicate boundary values: Handled gracefully by the predicate pattern —
> the affected middle partition produces an empty result set with no data
> loss.
> - Unsorted results: The connector wraps the boundary-query in a subquery
> with ORDER BY 1, so the user does not need to include ORDER BY.
> - Count mismatch: Same as 2.1 — fewer values reduce partition count; more
> values are rejected at validation.
> 
> Additionally, I added an Option Validation Matrix section to make the
> constraints across the three partitioning modes explicit in one place.
> 
> Please let me know if any of the updated content raises further questions.
> 
> Thank you again for your thorough review.
> 
> P.S Thank you for your interest Youngil Kim.
> 
> Best regards,
> ChanHae Oh
> 
> On Wed, Jul 8, 2026 at 2:48 PM Leonard Xu  wrote:
> 
> > Hi Chanhae,
> >
> > Thanks for driving this FLIP. The motivation makes sense to me: the current
> > equal-width JDBC partitioning does not work well for skewed data, and it
> > also
> > does not help much when users do not have a suitable numeric/date/timestamp
> > partition column.
> >
> > I am generally supportive of the direction, but I think both proposed
> > approaches
> > need clearer semantics.
> >
> > For physical-id partitioning, ROWID/ctid are physical identifiers rather
> > than
> > stable logical keys. They may change after updates, table rewrites,
> > vacuum, or
> > row movement. Since different splits may be read by different statements or
> > connections, the FLIP should clarify whether this mode only targets static
> > tables, or whether the connector can provide a consistent snapshot across
> > all
> > splits. Otherwise, missing or duplicated rows may be possible.
> >
> > For boundary-query partitioning, the contract should also be more precise.
> > For
> > example, if scan.partition.num is N, should the query return N - 1
> > boundaries?
> > How are inclusive/exclusive predicates generated? How should NULL values,
> > duplicate boundaries, unordered results, or boundary count mismatches be
> > handled?
> > These details are important to avoid missing or duplicated rows.
> >
> > It would also be helpful to define the option validation matrix clearly,
> > e.g.
> > which options are required or mutually exclusive for range partitioning,
> > boundary-query partitioning, and physical-id partitioning.
> >
> > Overall, I like the direction, but I think the FLIP should first nail down
> > the
> > correctness guarantees and edge-case behavior.
> >
> > Best,
> > Leonard
> 

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Chanhae Oh]

Hi Leonard,

Thank you for taking the time to review this proposal.
Your feedback touched on the exact boundary conditions and correctness
guarantees that I had not articulated precisely enough, and I genuinely
learned from the breadth of your perspective.

I have updated the FLIP to address each of your questions (changed sections
are highlighted in yellow).

1. Physical ID Partitioning — correctness under concurrent changes
(Updated: Proposed Changes > Physical ID Partitioning > Correctness
Guarantees and Limitations)

The two databases behave differently at the physical layer. Oracle ROWID is
stable under normal DML — it only changes under structural operations
(ALTER TABLE MOVE, EXPORT/IMPORT, row movement). PostgreSQL ctid, however,
changes on every UPDATE since a new tuple is written at a new location.
Physical-id partitioning for PostgreSQL is therefore only safe when no
concurrent DML runs during the scan.

2.1. Boundary-query — should the query return exactly N-1 values?
(Updated: Proposed Changes > Boundary Query > Partition count behavior)

Requiring exactly N-1 is too strict. Fewer values are allowed — the
partition count simply reduces to (returned values) + 1. More than N-1
values is a validation error, since scan.partition.num serves as a resource
cap and silently exceeding it could create more parallelism than the user
intended.

2.2. Boundary-query — how are inclusive/exclusive predicates generated?
(Updated: Proposed Changes > Boundary Query > Predicate contract)

Partition 1 uses col < v1 OR col IS NULL, middle partitions use v(i-1) <=
col < vi (left-inclusive, right-exclusive), and the last partition uses col
>= v(N-1). This ensures no row is emitted by two partitions simultaneously.

2.3. Boundary-query — NULL values, duplicate boundaries, unsorted results,
count mismatch
(Updated: Proposed Changes > Boundary Query > Predicate contract, Partition
count behavior)

- NULL: Routed to Partition 1 via the explicit col IS NULL clause, since
SQL NULL comparisons evaluate to NULL rather than TRUE.
- Duplicate boundary values: Handled gracefully by the predicate pattern —
the affected middle partition produces an empty result set with no data
loss.
- Unsorted results: The connector wraps the boundary-query in a subquery
with ORDER BY 1, so the user does not need to include ORDER BY.
- Count mismatch: Same as 2.1 — fewer values reduce partition count; more
values are rejected at validation.

Additionally, I added an Option Validation Matrix section to make the
constraints across the three partitioning modes explicit in one place.

Please let me know if any of the updated content raises further questions.

Thank you again for your thorough review.

P.S Thank you for your interest Youngil Kim.

Best regards,
ChanHae Oh

On Wed, Jul 8, 2026 at 2:48 PM Leonard Xu  wrote:

> Hi Chanhae,
>
> Thanks for driving this FLIP. The motivation makes sense to me: the current
> equal-width JDBC partitioning does not work well for skewed data, and it
> also
> does not help much when users do not have a suitable numeric/date/timestamp
> partition column.
>
> I am generally supportive of the direction, but I think both proposed
> approaches
> need clearer semantics.
>
> For physical-id partitioning, ROWID/ctid are physical identifiers rather
> than
> stable logical keys. They may change after updates, table rewrites,
> vacuum, or
> row movement. Since different splits may be read by different statements or
> connections, the FLIP should clarify whether this mode only targets static
> tables, or whether the connector can provide a consistent snapshot across
> all
> splits. Otherwise, missing or duplicated rows may be possible.
>
> For boundary-query partitioning, the contract should also be more precise.
> For
> example, if scan.partition.num is N, should the query return N - 1
> boundaries?
> How are inclusive/exclusive predicates generated? How should NULL values,
> duplicate boundaries, unordered results, or boundary count mismatches be
> handled?
> These details are important to avoid missing or duplicated rows.
>
> It would also be helpful to define the option validation matrix clearly,
> e.g.
> which options are required or mutually exclusive for range partitioning,
> boundary-query partitioning, and physical-id partitioning.
>
> Overall, I like the direction, but I think the FLIP should first nail down
> the
> correctness guarantees and edge-case behavior.
>
> Best,
> Leonard

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Leonard Xu]

Hi Chanhae,

Thanks for driving this FLIP. The motivation makes sense to me: the current
equal-width JDBC partitioning does not work well for skewed data, and it also
does not help much when users do not have a suitable numeric/date/timestamp
partition column.

I am generally supportive of the direction, but I think both proposed approaches
need clearer semantics.

For physical-id partitioning, ROWID/ctid are physical identifiers rather than
stable logical keys. They may change after updates, table rewrites, vacuum, or
row movement. Since different splits may be read by different statements or
connections, the FLIP should clarify whether this mode only targets static
tables, or whether the connector can provide a consistent snapshot across all
splits. Otherwise, missing or duplicated rows may be possible.

For boundary-query partitioning, the contract should also be more precise. For
example, if scan.partition.num is N, should the query return N - 1 boundaries?
How are inclusive/exclusive predicates generated? How should NULL values,
duplicate boundaries, unordered results, or boundary count mismatches be 
handled?
These details are important to avoid missing or duplicated rows.

It would also be helpful to define the option validation matrix clearly, e.g.
which options are required or mutually exclusive for range partitioning,
boundary-query partitioning, and physical-id partitioning.

Overall, I like the direction, but I think the FLIP should first nail down the
correctness guarantees and edge-case behavior. 

Best,
Leonard

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Youngil Kim]

Hi Chanhae

I fully support this proposal.

The current JDBC connector's limitation to numeric partition columns is a
significant pain point in practice. String-based partitioning via
scan.partition.column and the introduction of scan.partition.boundary-query
(inspired by Sqoop) are practical additions that directly address
real-world use cases.

In particular, the scan.partition.use-physical-id option for Oracle and
PostgreSQL using ROWID-based splitting is a clean approach to avoiding data
skew — something that has been difficult to handle reliably with the
existing strategy.

Looking forward to seeing this move forward. Happy to help with review or
testing if needed.

Best regards,
Young il Kim.


2026년 7월 8일 (수) 12:10, Chanhae Oh 님이 작성:

>  Hi Sergey, David,
>
> I hope this message finds you well.
> I am writing to ask if you would be willing to share your thoughts on a
> FLIP I have been working on — Extended JDBC Scan Partitioning Options for
> Efficient Data Ingestion.
>
> The proposal introduces two extensions to the JDBC connector's scan
> partitioning:
>
> 1. *scan.partition.use-physical-id* — For Oracle and PostgreSQL, which
> expose a physical row identifier (ROWID / ctid), partition bounds are
> derived directly from DB-native metadata. Because data is physically
> distributed across storage blocks, this produces naturally balanced splits
> without requiring any knowledge of the data distribution.
>
> 2. *scan.partition.boundary-query* — A SQL query returning ordered split
> points for the partition column. Unlike lower-bound / upper-bound, which
> divide a range into equal intervals regardless of actual distribution, this
> option lets users place split points where data is actually concentrated.
> It also extends support to string-type partition columns, which the current
> implementation does not handle.
>
> A discussion thread is already open on the dev mailing list, and the full
> draft is linked below.
>
>
>- JIRA: https://issues.apache.org/jira/browse/FLINK-36812
>- FLIP draft:
>
> https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing
>- Discussion:
>https://lists.apache.org/thread/wxh94hhyoy4b164m9q6cxhf8jypkpwdo
>
>
> Any feedback or suggestions would be greatly appreciated.
>
> Best regards,
> ChanHae Oh
> --------------
> *From:* Chanhae Oh 
> *Sent:* Tuesday, July 7, 2026 19:09
> *To:* Ryan van Huuksloot via dev 
> *Subject:* [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for
> Efficient Data Ingestion
>
> Hi all,
>
> Following up on FLINK-36812 [1], I've drafted a FLIP to address the
> limitations of the current scan.partition.column implementation.
> The existing approach splits data by equal numeric intervals, which breaks
> down under two common scenarios: skewed data distribution and the absence
> of a suitable numeric column (e.g., string-typed partition keys).
>
> Two extensions are proposed:
>
> *1. Physical ID Partitioning (scan.partition.use-physical-id)*
> For databases that expose a physical row identifier (Oracle ROWID,
> PostgreSQL ctid), the connector can derive partition bounds automatically
> from DB-native metadata.
> Since rows are physically distributed across storage blocks, this produces
> naturally balanced splits with no user-defined partition column required.
>
> *2. Boundary Query (scan.partition.boundary-query)*
> A SQL query returning ordered split points for the column specified in
> scan.partition.column. The connector uses these values to generate range
> predicates, enabling distribution-aware parallel reads for any column type
> — including strings.
> scan.partition.num is required and determines how many partitions are
> created.
>  This is conceptually close to Apache Sqoop's --boundary-query, extended to
> support string-type columns natively.
>
> Both extensions leave existing behavior unchanged.
>
> Full design: [2]
>
> Feedback and questions are welcome.
>
> Thanks,
> Chanhae Oh
>
> [1] https://issues.apache.org/jira/browse/FLINK-36812
> [2]
>
> https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing
>

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Chanhae Oh]

 Hi Sergey, David,

I hope this message finds you well.
I am writing to ask if you would be willing to share your thoughts on a
FLIP I have been working on — Extended JDBC Scan Partitioning Options for
Efficient Data Ingestion.

The proposal introduces two extensions to the JDBC connector's scan
partitioning:

1. *scan.partition.use-physical-id* — For Oracle and PostgreSQL, which
expose a physical row identifier (ROWID / ctid), partition bounds are
derived directly from DB-native metadata. Because data is physically
distributed across storage blocks, this produces naturally balanced splits
without requiring any knowledge of the data distribution.

2. *scan.partition.boundary-query* — A SQL query returning ordered split
points for the partition column. Unlike lower-bound / upper-bound, which
divide a range into equal intervals regardless of actual distribution, this
option lets users place split points where data is actually concentrated.
It also extends support to string-type partition columns, which the current
implementation does not handle.

A discussion thread is already open on the dev mailing list, and the full
draft is linked below.


   - JIRA: https://issues.apache.org/jira/browse/FLINK-36812
   - FLIP draft:
   
https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing
   - Discussion:
   https://lists.apache.org/thread/wxh94hhyoy4b164m9q6cxhf8jypkpwdo


Any feedback or suggestions would be greatly appreciated.

Best regards,
ChanHae Oh
--
*From:* Chanhae Oh 
*Sent:* Tuesday, July 7, 2026 19:09
*To:* Ryan van Huuksloot via dev 
*Subject:* [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for
Efficient Data Ingestion

Hi all,

Following up on FLINK-36812 [1], I've drafted a FLIP to address the
limitations of the current scan.partition.column implementation.
The existing approach splits data by equal numeric intervals, which breaks
down under two common scenarios: skewed data distribution and the absence
of a suitable numeric column (e.g., string-typed partition keys).

Two extensions are proposed:

*1. Physical ID Partitioning (scan.partition.use-physical-id)*
For databases that expose a physical row identifier (Oracle ROWID,
PostgreSQL ctid), the connector can derive partition bounds automatically
from DB-native metadata.
Since rows are physically distributed across storage blocks, this produces
naturally balanced splits with no user-defined partition column required.

*2. Boundary Query (scan.partition.boundary-query)*
A SQL query returning ordered split points for the column specified in
scan.partition.column. The connector uses these values to generate range
predicates, enabling distribution-aware parallel reads for any column type
— including strings.
scan.partition.num is required and determines how many partitions are
created.
 This is conceptually close to Apache Sqoop's --boundary-query, extended to
support string-type columns natively.

Both extensions leave existing behavior unchanged.

Full design: [2]

Feedback and questions are welcome.

Thanks,
Chanhae Oh

[1] https://issues.apache.org/jira/browse/FLINK-36812
[2]
https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[ChanBin Oh]

Hi Chanhae,

Thanks for sharing this. The limitations you mentioned about the current
equal-interval partitioning are very realistic, and both proposed
approaches make total sense. Huge +1 from me for this initiative!

Re: [DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[주수민]

+1 (non-binding)

Hi Chanhae,

The proposed approaches look very promising for handling skewed data
distributions in practice.

Looking forward to the FLIP progress.

Best,
Sumin Joo

2026년 7월 7일 (화) 오후 7:09, Chanhae Oh 님이 작성:

> Hi all,
>
> Following up on FLINK-36812 [1], I've drafted a FLIP to address the
> limitations of the current scan.partition.column implementation.
> The existing approach splits data by equal numeric intervals, which breaks
> down under two common scenarios: skewed data distribution and the absence
> of a suitable numeric column (e.g., string-typed partition keys).
>
> Two extensions are proposed:
>
> 1. Physical ID Partitioning (scan.partition.use-physical-id)
> For databases that expose a physical row identifier (Oracle ROWID,
> PostgreSQL ctid), the connector can derive partition bounds automatically
> from DB-native metadata.
> Since rows are physically distributed across storage blocks, this produces
> naturally balanced splits with no user-defined partition column required.
>
> 2. Boundary Query (scan.partition.boundary-query)
> A SQL query returning ordered split points for the column specified in
> scan.partition.column. The connector uses these values to generate range
> predicates, enabling distribution-aware parallel reads for any column type
> — including strings.
> scan.partition.num is required and determines how many partitions are
> created.
>  This is conceptually close to Apache Sqoop's --boundary-query, extended
> to support string-type columns natively.
>
> Both extensions leave existing behavior unchanged.
>
> Full design: [2]
>
> Feedback and questions are welcome.
>
> Thanks,
> Chanhae Oh
>
> [1] https://issues.apache.org/jira/browse/FLINK-36812
> [2]
> https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing
>
>

[DISCUSS] FLIP-XXX: Extended JDBC Scan Partitioning Options for Efficient Data Ingestion

[Chanhae Oh]

Hi all,

Following up on FLINK-36812 [1], I've drafted a FLIP to address the limitations 
of the current scan.partition.column implementation.
The existing approach splits data by equal numeric intervals, which breaks down 
under two common scenarios: skewed data distribution and the absence of a 
suitable numeric column (e.g., string-typed partition keys).

Two extensions are proposed:

1. Physical ID Partitioning (scan.partition.use-physical-id)
For databases that expose a physical row identifier (Oracle ROWID, PostgreSQL 
ctid), the connector can derive partition bounds automatically from DB-native 
metadata.
Since rows are physically distributed across storage blocks, this produces 
naturally balanced splits with no user-defined partition column required.

2. Boundary Query (scan.partition.boundary-query)
A SQL query returning ordered split points for the column specified in 
scan.partition.column. The connector uses these values to generate range 
predicates, enabling distribution-aware parallel reads for any column type — 
including strings.
scan.partition.num is required and determines how many partitions are created.
 This is conceptually close to Apache Sqoop's --boundary-query, extended to 
support string-type columns natively.

Both extensions leave existing behavior unchanged.

Full design: [2]

Feedback and questions are welcome.

Thanks,
Chanhae Oh

[1] https://issues.apache.org/jira/browse/FLINK-36812
[2] 
https://docs.google.com/document/d/1fgnYuOo9U6PPuwJgvnmdfGfV70hhO2xBTSo0OR46r1g/edit?usp=sharing