[
https://issues.apache.org/jira/browse/KAFKA-21085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114758#comment-18114758
]
sanghyeok An commented on KAFKA-21085:
--------------------------------------
[~lianetm] , [~kirktrue]
Hi! Sorry for the pinging you out of the blue...!
I was looking through the async consumer’s poll code and found a case that I
wanted to ask you about.
When an AsyncPollEvent is waiting for an OffsetFetch response, the auto-commit
timer can expire and make maximumTimeToWait() return 0. The application thread
then keeps looping without waiting. But since the existing event hasn’t
completed, it can’t submit another AsyncPollEvent, so the auto-commit check
doesn’t run again.
I reproduced this in a local diagnostic test, though I haven’t checked the
actual CPU impact in a running cluster yet. At first, I thought we could just
wait for a while when an event is still pending. But the more I looked at it,
the less sure I was about which timeout to use. Waiting for the remaining poll
timeout could delay noticing that the event has completed, and retry.backoff.ms
can also be set quite high.
For example, suppose an AsyncPollEvent is still pending and the wait times are:
* Auto-commit manager: 0ms, because the auto-commit timer has expired.
* Other managers: 10ms as their minimum wait limit.
* retry.backoff.ms: 15000ms.
* Remaining poll timeout: 30000ms.
In this case, applicationEventHandler.maximumTimeToWait() returns 0. If we
replace that with the retry backoff or the remaining poll timeout, we could end
up waiting 15 or 30 seconds, even though another manager returned a limit of
10ms. Heartbeats may still be sent by the background thread, but I’m concerned
that this could delay the application thread’s handling of group changes or
other background events. That’s why I’m not sure simply replacing the zero
timeout with a backoff is the right fix..
I’m also wondering how much this matters in practice. If it only happens
briefly during startup, perhaps it isn’t worth changing the current behavior.
But if it can happen repeatedly during normal operation, such as after
partition reassignment, it seems worth looking into. Does this look like
something we should address? I’d be interested to hear how you think this
should work with the current event lifecycle.
Thanks for taking a look!
I can share the diagnostic test if that would help.
> Async consumer busy-spins when auto-commit expires during offset
> initialization
> -------------------------------------------------------------------------------
>
> Key: KAFKA-21085
> URL: https://issues.apache.org/jira/browse/KAFKA-21085
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
>
> AsyncKafkaConsumer.poll() can busy-spin when the auto-commit interval expires
> while offset initialization is still waiting for an OffsetFetch response.
>
> The pending initialization keeps the current AsyncPollEvent incomplete,
> preventing another AsyncPollEvent from being submitted. With the coordinator
> known and no auto-commit in flight, the expired auto-commit timer causes
> CommitRequestManager.maximumTimeToWait() to return 0.
>
> Consequently, when no fetch can be returned, pollForFetches() repeatedly
> calls FetchBuffer.awaitWakeup() with a zero timeout. The auto-commit check
> that would reset the timer is not executed again because no new
> AsyncPollEvent is processed.
>
> This exposes a mismatch between wait-time calculation and poll-event
> processing: the application thread is repeatedly told not to wait, although
> it cannot trigger the auto-commit check again. Other paths may expose a
> similar mismatch, but have not yet been
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
