Hongshun Wang created FLINK-41564:
-------------------------------------

             Summary: Improve bucket assignment balance across partitions in 
Fluss source
                 Key: FLINK-41564
                 URL: https://issues.apache.org/jira/browse/FLINK-41564
             Project: Flink
          Issue Type: New Feature
          Components: Flink CDC
    Affects Versions: cdc-3.7.0
            Reporter: Hongshun Wang
             Fix For: cdc-3.8.0


{{FlinkSourceEnumerator#getSplitOwner()}} currently calculates a separate 
starting subtask for each partition using its partition ID:

 
{code:java}
startIndex = ((partitionId.hashCode() * 31) & 0x7FFFFFFF) % parallelism;
owner = (startIndex + bucketId) % parallelism;
{code}
 

This does not guarantee balanced bucket assignment across partitions. Partition 
IDs are globally allocated and may be sparse within a table. Multiple 
partitions can therefore have the same starting subtask, leaving some readers 
overloaded while others are idle.

For example, with source parallelism {{{}4{}}}, one bucket per partition, and 
partition IDs {{{}0, 4, 8, 12{}}}, the current bucket distribution is:

 
{code:java}
Current:  [4, 0, 0, 0]
Expected: [1, 1, 1, 1]
{code}
 

 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to