beomjungo created KAFKA-21590:
---------------------------------

             Summary: RoundRobinPartitioner starts every producer's counter at 
0, so producers started together write to the same partitions in lockstep
                 Key: KAFKA-21590
                 URL: https://issues.apache.org/jira/browse/KAFKA-21590
             Project: Kafka
          Issue Type: Improvement
          Components: clients, producer 
    Affects Versions: 4.2.1, 3.9.2
            Reporter: beomjungo


h3. Problem
{{RoundRobinPartitioner}} creates its per-topic counter with {{new 
AtomicInteger(0)}} (trunk: RoundRobinPartitioner.java, {{nextValue}}). Every 
producer instance therefore starts from the first entry of 
{{availablePartitionsForTopic}}, and all producers in the same cluster see the 
same partition list.

When several producers start at the same time (e.g. a Kubernetes rollout) and 
each sends the same number of records, their counters stay equal. The k-th 
record of every producer goes to the same partition. Instead of spreading load, 
N producers stack N records on the same few partitions.

Round robin is only guaranteed within one producer, so this is not a 
correctness bug. However, it makes the multi-producer result worse than the 
default sticky partitioner, which picks a random partition 
({{ThreadLocalRandom}}).

h3. Reproduction (kafka-clients 3.9.2 and 4.2.1)
{code:java}
Node node = new Node(0, "localhost", 9092);
List<PartitionInfo> infos = new ArrayList<>();
for (int p = 0; p < 216; p++)
    infos.add(new PartitionInfo("t", p, node, new Node[]{node}, new 
Node[]{node}));
Cluster cluster = new Cluster("c", List.of(node), infos, Set.of(), Set.of());

// 27 producers (one partitioner each) started together, 4 records each
Map<Integer, Integer> perPartition = new TreeMap<>();
for (int i = 0; i < 27; i++) {
    RoundRobinPartitioner rr = new RoundRobinPartitioner();
    for (int k = 0; k < 4; k++)
        perPartition.merge(rr.partition("t", null, null, null, null, cluster), 
1, Integer::sum);
}
System.out.println(perPartition);
// 3.9.2 and 4.2.1: {0=27, 1=27, 2=27, 3=27}  -> 108 records on 4 of 216 
partitions
{code}

h3. Observed in production
* 27 consumer pods each re-publish fan-out pieces to a 216-partition topic with 
{{partitioner.class=RoundRobinPartitioner}}.
* An upstream round-robin step delivers the same number of pieces to every pod, 
so the counters never drift apart.
* Over 4 hours after a rollout, 2,142 records landed on only 82 of 216 
partitions, 77 of them with exactly 27 records (= number of pods). 134 
partitions received nothing.
* Partition skew (max / average inflow) rose from 7.9 with the default sticky 
partitioner to 14.4 after switching to RoundRobinPartitioner.

h3. Proposed change
Start each counter at a random value, as the pre-2.4 {{DefaultPartitioner}} did 
({{new AtomicInteger(ThreadLocalRandom.current().nextInt())}}):
{code:java}
AtomicInteger counter = topicCounterMap.computeIfAbsent(topic,
        k -> new AtomicInteger(ThreadLocalRandom.current().nextInt()));
{code}
* No need to know the partition count up front: {{partition()}} already applies 
{{Utils.toPositive(next) % availablePartitions.size()}} on every call.
* Round robin within one producer is unchanged. Only the starting partition 
becomes random.
* In a replay of the same production traffic (27 producers, 216 partitions), 
the max/ideal ratio per event drops from 15.8 to 2.2. 1.0 would be perfectly 
even.

h3. Prior art
* KAFKA-1071 fixed the same issue for the old producer: its partition counter 
started at 0, so producers started at the same time tended to pick the same 
partition.
* {{DefaultPartitioner}} up to 2.3 initialized its counter with 
{{ThreadLocalRandom.current().nextInt()}}. {{RoundRobinPartitioner}} (KIP-369, 
2.4) did not carry this over.

h3. Related issues
Related but different: KAFKA-9965, KAFKA-16283, KAFKA-17632 and KAFKA-14573 are 
about partitions being skipped within one producer. This issue is about 
multiple producers being in sync.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to