beomjungo created KAFKA-21590:
---------------------------------
Summary: RoundRobinPartitioner starts every producer's counter at
0, so producers started together write to the same partitions in lockstep
Key: KAFKA-21590
URL: https://issues.apache.org/jira/browse/KAFKA-21590
Project: Kafka
Issue Type: Improvement
Components: clients, producer
Affects Versions: 4.2.1, 3.9.2
Reporter: beomjungo
h3. Problem
{{RoundRobinPartitioner}} creates its per-topic counter with {{new
AtomicInteger(0)}} (trunk: RoundRobinPartitioner.java, {{nextValue}}). Every
producer instance therefore starts from the first entry of
{{availablePartitionsForTopic}}, and all producers in the same cluster see the
same partition list.
When several producers start at the same time (e.g. a Kubernetes rollout) and
each sends the same number of records, their counters stay equal. The k-th
record of every producer goes to the same partition. Instead of spreading load,
N producers stack N records on the same few partitions.
Round robin is only guaranteed within one producer, so this is not a
correctness bug. However, it makes the multi-producer result worse than the
default sticky partitioner, which picks a random partition
({{ThreadLocalRandom}}).
h3. Reproduction (kafka-clients 3.9.2 and 4.2.1)
{code:java}
Node node = new Node(0, "localhost", 9092);
List<PartitionInfo> infos = new ArrayList<>();
for (int p = 0; p < 216; p++)
infos.add(new PartitionInfo("t", p, node, new Node[]{node}, new
Node[]{node}));
Cluster cluster = new Cluster("c", List.of(node), infos, Set.of(), Set.of());
// 27 producers (one partitioner each) started together, 4 records each
Map<Integer, Integer> perPartition = new TreeMap<>();
for (int i = 0; i < 27; i++) {
RoundRobinPartitioner rr = new RoundRobinPartitioner();
for (int k = 0; k < 4; k++)
perPartition.merge(rr.partition("t", null, null, null, null, cluster),
1, Integer::sum);
}
System.out.println(perPartition);
// 3.9.2 and 4.2.1: {0=27, 1=27, 2=27, 3=27} -> 108 records on 4 of 216
partitions
{code}
h3. Observed in production
* 27 consumer pods each re-publish fan-out pieces to a 216-partition topic with
{{partitioner.class=RoundRobinPartitioner}}.
* An upstream round-robin step delivers the same number of pieces to every pod,
so the counters never drift apart.
* Over 4 hours after a rollout, 2,142 records landed on only 82 of 216
partitions, 77 of them with exactly 27 records (= number of pods). 134
partitions received nothing.
* Partition skew (max / average inflow) rose from 7.9 with the default sticky
partitioner to 14.4 after switching to RoundRobinPartitioner.
h3. Proposed change
Start each counter at a random value, as the pre-2.4 {{DefaultPartitioner}} did
({{new AtomicInteger(ThreadLocalRandom.current().nextInt())}}):
{code:java}
AtomicInteger counter = topicCounterMap.computeIfAbsent(topic,
k -> new AtomicInteger(ThreadLocalRandom.current().nextInt()));
{code}
* No need to know the partition count up front: {{partition()}} already applies
{{Utils.toPositive(next) % availablePartitions.size()}} on every call.
* Round robin within one producer is unchanged. Only the starting partition
becomes random.
* In a replay of the same production traffic (27 producers, 216 partitions),
the max/ideal ratio per event drops from 15.8 to 2.2. 1.0 would be perfectly
even.
h3. Prior art
* KAFKA-1071 fixed the same issue for the old producer: its partition counter
started at 0, so producers started at the same time tended to pick the same
partition.
* {{DefaultPartitioner}} up to 2.3 initialized its counter with
{{ThreadLocalRandom.current().nextInt()}}. {{RoundRobinPartitioner}} (KIP-369,
2.4) did not carry this over.
h3. Related issues
Related but different: KAFKA-9965, KAFKA-16283, KAFKA-17632 and KAFKA-14573 are
about partitions being skipped within one producer. This issue is about
multiple producers being in sync.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)