[
https://issues.apache.org/jira/browse/KAFKA-18376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianet Magrans updated KAFKA-18376:
-----------------------------------
Labels: consumer-threading-refactor (was: )
> High CPU load when AsyncKafkaConsumer uses a small max poll value
> -----------------------------------------------------------------
>
> Key: KAFKA-18376
> URL: https://issues.apache.org/jira/browse/KAFKA-18376
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Reporter: Philip Nee
> Assignee: Philip Nee
> Priority: Major
> Labels: consumer-threading-refactor
>
> We stress tested the AsyncConsumer using maxPoll = 5 and observed abnormally
> high CPU usage. Under normal usage (with defaults), the consumers on average
> use around 10% of the CPU with 20mb/s byte rate, which is aligned with the
> ClassicKafkaConsumer. As we tested the consumer with a small max poll value,
> we observed the CPU usage spikes to > 50% while the classic consumer stays at
> around 10%.
>
> _note: percentage of CPU usage may depend on the running pod hardware._
>
> The profiling results shows two major contributors to the CPU cycles
> # AsyncKafkaConsumer.updateFetchPosition (addAndGet & new
> CheckAndUpdatePositionEvent())
> # AbstractFetch.fetchablePartitions from the fetchrequestmanager
>
> for AsyncKafkaConsumer.updateFetchPosition - It seems like
> * UUID generation can become quite expensive. This is particularly
> noticeable when creating large number of events
> * ConsumerUtils.getResult, which uses future.get() also consumes quite a bit
> of CPU cycles
> for fetchablePartitions, FetchBuffer.bufferedPartitions which uses Java
> ConcurrentLinkedQueue.forEach also consumes quite a bit of CPUs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
