[
https://issues.apache.org/jira/browse/FLINK-41564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41564:
-----------------------------------
Labels: pull-request-available (was: )
> Improve bucket assignment balance across partitions in Fluss source
> -------------------------------------------------------------------
>
> Key: FLINK-41564
> URL: https://issues.apache.org/jira/browse/FLINK-41564
> Project: Flink
> Issue Type: New Feature
> Components: Flink CDC
> Affects Versions: cdc-3.7.0
> Reporter: Hongshun Wang
> Assignee: Hongshun Wang
> Priority: Major
> Labels: pull-request-available
> Fix For: cdc-3.8.0
>
>
> {{FlinkSourceEnumerator#getSplitOwner()}} currently calculates a separate
> starting subtask for each partition using its partition ID:
>
> {code:java}
> startIndex = ((partitionId.hashCode() * 31) & 0x7FFFFFFF) % parallelism;
> owner = (startIndex + bucketId) % parallelism;
> {code}
>
> This does not guarantee balanced bucket assignment across partitions.
> Partition IDs are globally allocated and may be sparse within a table.
> Multiple partitions can therefore have the same starting subtask, leaving
> some readers overloaded while others are idle.
> For example, with source parallelism {{{}4{}}}, one bucket per partition, and
> partition IDs {{{}0, 4, 8, 12{}}}, the current bucket distribution is:
>
> {code:java}
> Current: [4, 0, 0, 0]
> Expected: [1, 1, 1, 1]
> {code}
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)