[
https://issues.apache.org/jira/browse/KAFKA-19296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-19296:
------------------------------
Description: The {{AsyncKafkaConsumer}} uses an event queue-based mechanism
to communicate between the application thread and the background network
thread. When a new event is enqueued, the application thread will call
{{NetworkClient.wakeup()}} in order to make sure the background thread doesn't
block in {{NetworkClient.poll()}} too long. However, in certain settings, the
number of wake ups for the {{AsyncKafkaConsumer}} can be a one, possibly two
orders of magnitude more than {{{}ClassicKafkaConsumer{}}}. Because
{{NetworkClient.wakeup()}} needs to call down through the OS' network layer,
the current event notification approach used by {{AsyncKafkaConsumer}} adds
significant, unnecessary overhead. (was: The {{AsyncKafkaConsumer}} uses an
event queue-based mechanism to communicate between the application thread and
the background network thread. Unique IDs were added to the events, mostly as a
way to disambiguate different events in logs. However, in situations where many
events are being created in a tight loop (e.g. polling in the consumer a few
records at a time), there is an inordinate amount of CPU time spent achieving
the randomness for the event IDs. Since there is no functional benefit to
having IDs for events, and little likelihood of individual events being
included in logs, ID generation should be removed.)
> Revise AsyncKafkaConsumer event wakeup approach
> -----------------------------------------------
>
> Key: KAFKA-19296
> URL: https://issues.apache.org/jira/browse/KAFKA-19296
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
> Reporter: Kirk True
> Assignee: Kirk True
> Priority: Major
> Labels: consumer-threading-refactor, performance
> Fix For: 4.1.0
>
>
> The {{AsyncKafkaConsumer}} uses an event queue-based mechanism to communicate
> between the application thread and the background network thread. When a new
> event is enqueued, the application thread will call
> {{NetworkClient.wakeup()}} in order to make sure the background thread
> doesn't block in {{NetworkClient.poll()}} too long. However, in certain
> settings, the number of wake ups for the {{AsyncKafkaConsumer}} can be a one,
> possibly two orders of magnitude more than {{{}ClassicKafkaConsumer{}}}.
> Because {{NetworkClient.wakeup()}} needs to call down through the OS' network
> layer, the current event notification approach used by {{AsyncKafkaConsumer}}
> adds significant, unnecessary overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
