[
https://issues.apache.org/jira/browse/KAFKA-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974615#comment-16974615
]
Chris Pettitt commented on KAFKA-9191:
--------------------------------------
We have not teased that apart. We do have the broker metrics available to look
at (and they can be regenerated with one or both of the in-house perf tools).
On the broker side we observed heavier CPU utilization with 100 byte messages
(capping at 100%), while utilization was lower (50-75%) for 512 byte messages.
> Kafka throughput suffers substantially when scaling topic partitions with
> small messages
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-9191
> URL: https://issues.apache.org/jira/browse/KAFKA-9191
> Project: Kafka
> Issue Type: Bug
> Reporter: Chris Pettitt
> Priority: Major
>
> We have observed, using two entirely different tools, that a simple Kafka
> application (read 1 topic and immediately produce to another) suffers
> substantial throughput degradation when scaling up topics. Below is the
> output of one of these tools, showing that going from 1 partition to 1000
> partitions yields a ~30% throughput decrease when messages are 100 bytes long.
> Using the same two tools, we observed that increasing the message size to 512
> bytes yields a throughput increase of ~20% going from 1 topic partition to
> 1000 topic partitions with all other variables held constant.
>
> |Kafka Core Testing| | | | | | | | | | |
> |Enable Transaction|Batch Size (b)|Linger (ms)|Max Inflight|Commit Interval
> (ms)|Num Records|Record Size (b)|Num Input Topics|1 Partition MB/s|1000
> Partitions MB/s|MB/s delta|
> |FALSE|16384|100|5|1000|20000000|100|1|45.633625|31.482193|-31.01%|
> |FALSE|16384|100|5|1000|20000000|512|1|70.217902|85.319107|21.51%|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
