Any update on this? Here’s what we’ve tried so far:
We briefly looked at the code and here's what we understand: Kafka server
accepts a a batch of messages for each topic partition. Map[TopicAndPartition,
It tries to process the entire batch within the configured timeout, and sends
back a response containing successful and failed records.
Now let's look at what can go wrong on a topic level, and how that would affect
the entire batch:
1) Not all replicas available for a topic: Let's say a batch contains data for
three topics: test1, test2, test3. For test3, not all replicas are available.
This case will be detected by the leader instantly, and it will fail the
records. In this case, the batch won't be delayed, and throughput for test1,
and test2 won't be affected.
2) Replicas for a topic are slow. Let's say that the replicas for only test3
are slow. In this case, replicating messages for test3 will take time, which in
turn will delay the entire batch, thus affecting the throughput of topics test1
What else can go wrong?
From: Ninad Naik <ninad.n...@epsilon.com<mailto:ninad.n...@epsilon.com>>
Date: Friday, August 4, 2017 at 12:45 PM
Subject: KafkaProducer (0.9): Single producer multiple topics.
We’re currently using one producer per topic. We’re considering switching to
one producer for multiple topics for obvious reasons.
Now, is there a case where one topic might be slower for some reason, and not
the others? And if there is such a case, how would it play out with a single
producer producing to multiple topics? Would it affect throughput of other