ableegoldman opened a new pull request #11085: URL: https://github.com/apache/kafka/pull/11085
While investigating some system test failures I found the Streams logs often got up to several GB in size, with much of it being due to these three lines being logged on repeat many times per ms: ``` [2021-07-19 11:15:26,880] DEBUG [] Invoking poll on main Consumer [2021-07-19 11:15:26,880] DEBUG [] Main Consumer poll completed in 0 ms and fetched 0 records [2021-07-19 11:15:26,880] DEBUG [] Thread state is already PENDING_SHUTDOWN, skipping the run once call ``` Even though they're debug logs this is just way too much printed way too often, and also reveals how much cpu we're wasting just spinning in this busy loop doing nothing else. It seems the situation was that the thread had been told to shut down but was waiting for an in-progress rebalance to complete before it could exit the poll loop and complete its shutdown. Turns out we were polling with `Duration.ZERO` in this case and therefore just running through the loop again and again and again within a single millisecond. So, we should be passing in a larger timeout to `poll()` when the thread is in `PENDING_SHUTDOWN` or `PENDING_ERROR` to let it actually wait for the ongoing rebalance to finish. Also threw in a few miscellaneous fixes/cleanups that I came across on the side -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
