m1a2st opened a new pull request, #23014:
URL: https://github.com/apache/kafka/pull/23014
`FetchRequestManager` currently wakes up the `FetchBuffer` unconditionally
whenever `prepareFetchRequests()` returns no requests to send. This behavior
was intended for one specific case: all currently fetchable partitions
already
have buffered data waiting to be consumed. However, the wakeup also happens
for
any other case that produces an empty result, including states where no
progress
can be made until some external event occurs:
- No partitions are fetchable at all (no assignment yet, invalid positions,
paused, pending revocation/callback).
- The partition's leader is unknown (`metadata.requestUpdate()` was just
triggered).
- The target node is in its reconnect backoff window.
- The target node already has an in-flight fetch request outstanding.
- The target node hosts other partitions that already have buffered data.
Because this wakeup bypasses the caller's normal `retryBackoffMs` delay, the
application thread and the background `ConsumerNetworkThread` can enter a
tight
loop: `poll()` issues a fetch-request event, the background thread finds
nothing
to send, wakes the buffer, `poll()` returns immediately, and the cycle
repeats
without any meaningful delay.
This is a pre-existing issue introduced with the non-blocking
`AsyncPollEvent` redesign in KAFKA-18376. However, KIP-909's async bootstrap
DNS
resolution extends the "no node available" window from effectively
instantaneous
to up to `bootstrapResolveTimeoutMs`, making the busy loop much more visible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
