[ 
https://issues.apache.org/jira/browse/FLINK-41564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-41564:
-----------------------------------
    Labels: pull-request-available  (was: )

> Improve bucket assignment balance across partitions in Fluss source
> -------------------------------------------------------------------
>
>                 Key: FLINK-41564
>                 URL: https://issues.apache.org/jira/browse/FLINK-41564
>             Project: Flink
>          Issue Type: New Feature
>          Components: Flink CDC
>    Affects Versions: cdc-3.7.0
>            Reporter: Hongshun Wang
>            Assignee: Hongshun Wang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: cdc-3.8.0
>
>
> {{FlinkSourceEnumerator#getSplitOwner()}} currently calculates a separate 
> starting subtask for each partition using its partition ID:
>  
> {code:java}
> startIndex = ((partitionId.hashCode() * 31) & 0x7FFFFFFF) % parallelism;
> owner = (startIndex + bucketId) % parallelism;
> {code}
>  
> This does not guarantee balanced bucket assignment across partitions. 
> Partition IDs are globally allocated and may be sparse within a table. 
> Multiple partitions can therefore have the same starting subtask, leaving 
> some readers overloaded while others are idle.
> For example, with source parallelism {{{}4{}}}, one bucket per partition, and 
> partition IDs {{{}0, 4, 8, 12{}}}, the current bucket distribution is:
>  
> {code:java}
> Current:  [4, 0, 0, 0]
> Expected: [1, 1, 1, 1]
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to