[
https://issues.apache.org/jira/browse/KAFKA-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianet Magrans updated KAFKA-20915:
-----------------------------------
Description:
On the async consumer, the fetch req manager has logic to wake up the fetch
buffer if it doesn't generate any fetch request to send.
[https://github.com/apache/kafka/blob/dacff9c85a29806fcafe617feb2db0c9677e6de7/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchRequestManager.java#L149-L151]
That wake up is needed in some cases (to unblock app thread if the case was no
fetch req generated because partitions were paused, not subscribed etc)., but
the wakeup is useless and harmful if the case was that no requests was
generated because they were already in-flight.
* Useless because nothing has changed yet regarding data (it is the fetch
response for the inflights that should wake up the app thread)
* harmful because it leads to a spin (high CPU) between the app thread and the
background (background wakes ups app thread when nothing really changed
regarding data, we're just waiting for in-flight requests that will themselves
wake up the buffer when complete).
This wakeup behaviour was introduced in 4.2, but it's impact on CPU due to
unneeded app thread - background thread spin was revealed with a recent
correctness fix https://issues.apache.org/jira/browse/KAFKA-20780
was:
On the async consumer, the fetch req manager has logic to wake up the fetch
buffer if it doesn't generate any fetch request to send.
[https://github.com/apache/kafka/blob/dacff9c85a29806fcafe617feb2db0c9677e6de7/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchRequestManager.java#L149-L151]
That wake up is needed in some cases (to unblock app thread if the case was no
fetch req generated because partitions were paused, not subscribed etc)., but
the wakeup is useless and harmful if the case was that no requests was
generated because they were already in-flight.
* Useless because nothing has changed yet regarding data (it is the fetch
response for the inflights that should wake up the app thread)
* harmful because it leads to a spin (high CPU) between the app thread and the
background (background wakes ups app thread when nothing really changed
regarding data, we're just waiting for in-flight requests that will themselves
wake up the buffer then complete).
This wakeup behaviour was introduced in 4.2, but it's impact on CPU due to
unneeded app thread - background thread spin was revealed with a recent
correctness fix https://issues.apache.org/jira/browse/KAFKA-20780
> Unneeded app thread wakeup when a fetch already in flight and nothing has
> changed in AsyncConsumer
> --------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20915
> URL: https://issues.apache.org/jira/browse/KAFKA-20915
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Reporter: Lianet Magrans
> Assignee: Lianet Magrans
> Priority: Blocker
> Fix For: 4.4.0
>
>
> On the async consumer, the fetch req manager has logic to wake up the fetch
> buffer if it doesn't generate any fetch request to send.
> [https://github.com/apache/kafka/blob/dacff9c85a29806fcafe617feb2db0c9677e6de7/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchRequestManager.java#L149-L151]
>
> That wake up is needed in some cases (to unblock app thread if the case was
> no fetch req generated because partitions were paused, not subscribed etc).,
> but the wakeup is useless and harmful if the case was that no requests was
> generated because they were already in-flight.
> * Useless because nothing has changed yet regarding data (it is the fetch
> response for the inflights that should wake up the app thread)
> * harmful because it leads to a spin (high CPU) between the app thread and
> the background (background wakes ups app thread when nothing really changed
> regarding data, we're just waiting for in-flight requests that will
> themselves wake up the buffer when complete).
> This wakeup behaviour was introduced in 4.2, but it's impact on CPU due to
> unneeded app thread - background thread spin was revealed with a recent
> correctness fix https://issues.apache.org/jira/browse/KAFKA-20780
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
