[
https://issues.apache.org/jira/browse/BEAM-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933334#comment-16933334
]
TJ commented on BEAM-8121:
--------------------------
[~aromanenko] i've done similar kind of similar tests before with reading from
one topic.
*Without* Reshuffle:
* Read from Kafka 3200message/s
* Read from Kafka write directly to BigQuery ± the same 3200message/s
* Read from Kafka do data transformations *without* writing to BigQuery
1500message/s
* Read from Kafka do data transformations *with* writing to BigQuery
1500message/s
Message consumption rate was not changing when machine number was increasing.
So my conclusions were that slow pipeline transformation steps decreases
general throughput. So I assume its fusion issue?
*With* Reshuffle:
* Read from Kafka do data transformations and write to BigQuery. With one
machine it was slower 1000message/s. When machine number increased to 10 then
it was almost the same message read rate 3100 message/s
When reading from many topics with reshuffle step consumption was much slower
450message/s and scaling was slow. CPU was hitting 100%. When scaled to 5
machines it only was able to read 1500msg/s. I will try to add more powerful
machines and retry reshuffling with it.
> Messages are not distributed per machines when consuming from Kafka topic
> with 1 partition
> ------------------------------------------------------------------------------------------
>
> Key: BEAM-8121
> URL: https://issues.apache.org/jira/browse/BEAM-8121
> Project: Beam
> Issue Type: Bug
> Components: io-java-kafka
> Affects Versions: 2.14.0
> Reporter: TJ
> Priority: Major
> Attachments: datalake-dataflow-cleaned.zip
>
>
> Messages are consumed from Kafka using KafkaIO. Each kafka topic contains
> only 1 partition. (That means that messages can be consumed only by one
> Consumer per 1 consumer group)
> When backlog of topic grows and system scales from 1 to X machines, all the
> messages seems to be executed on the same machine on which they are read.
> Due to that message throughput doesn't increase comparing X machines to 1
> machine. If one machine was reading 2K messages per s, X machines will be
> reading the same amount.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
