lianetm opened a new pull request, #23124: URL: https://github.com/apache/kafka/pull/23124
The async consumer wakes up the buffer when adding data to it (all good, unchanged with this PR, unblocks the app thread to collect the data). But it also wakes the buffer when it attempts to generate fetch requests and it can't generate any. This included the case when fetch request was already in-flight (so can't generate any more fetch requests). This leads to a spin between the 2 threads: 1. application thread finds the buffer empty and blocks 2. background thread cannot generate new fetch (one already in-flight), wakes the buffer, completes the event 3. application thread wakes immediately, still has nothing to collect, submits another poll event No blocking and no backoff, so spinning generating and completing poll events for as long as a request/response is inflight. This PR includes a fix to ensure the explicit wakeup triggered when no requests can be generated fires only when it can achieve something (nothing is in flight, and some partition is fetchable). It also includes a fix to ensure that every in-flight request wakes the buffer when it completes, no matter the outcome. This ensures progress in all cases and aligns with the classic consumer (it blocks on the network poll so effectively unblocks on any fetch outcome) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
