Chris Pettitt created KAFKA-9191:
------------------------------------
Summary: Kafka throughput suffers substantially when scaling topic
partitions with small messages
Key: KAFKA-9191
URL: https://issues.apache.org/jira/browse/KAFKA-9191
Project: Kafka
Issue Type: Bug
Reporter: Chris Pettitt
We have observed, using two entirely different tools, that a simple Kafka
application (read 1 topic and immediately produce to another) suffers
substantial performance degradation when scaling up topics. Below is the output
of one of these tools, showing that going from 1 partition to 1000 partitions
yields a ~30% throughput decrease when messages are 100 bytes long.
Using the same two tools, we observed that increasing the message size to 512
bytes yields a throughput increase of ~20% going from 1 topic partition to 1000
topic partitions with all other variables held constant.
|Kafka Core Testing| | | | | | | | | | |
|Enable Transaction|Batch Size (b)|Linger (ms)|Max Inflight|Commit Interval
(ms)|Num Records|Record Size (b)|# Input Topics|1 Partition MB/s|1000
Partitions MB/s|MB/s delta|
|FALSE|16384|100|5|1000|20000000|100|1|45.633625|31.482193|-31.01%|
|FALSE|16384|100|5|1000|20000000|512|1|70.217902|85.319107|21.51%|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
