harmadasg commented on PR #23854: URL: https://github.com/apache/kafka/pull/23854#issuecomment-6295598089
The recent failures in the newly added `testDynamicThreadCountsAreAppliedAfterRestart` uncovered an existing bug outside of my recent changes. In short, shutdown can hang if it starts right after a request handler thread-pool shrink event. (num.io.threads) 1. KafkaRequestHandlerPool.resizeThreadPool shrinks the pool by calling stop() on the removed handlers 2. A stopped handler only exits after its current receiveRequest(300) poll returns. 3. If shutdown() runs during that window, it sends one ShutdownRequest per remaining handler 4. A removed handler can take one of those requests and exit, so a remaining handler never gets one and awaitShutdown() blocks forever This scenario is not likely to happen on a real broker as the full shutdown process takes multiple seconds and the shutdown has to start within ~300ms of a shrink. But still it can affect existing integration tests. I pushed e7b3f86b97ddf953f7b2ef081d4551c545c0045f to fix the shutdown race. Also added a new unit test to verify the change. As mentioned the bug exists on trunk independently, the new test just makes it reachable so the new test should fail against trunk. The test is not 100% deterministic but shrinking the pool from 16 to 1 and then shutdown has a high chance that the shutdown request will find one of the 15 removed handlers. While working on a fix I also noticed another problematic part where filtering for to be removed handlers can prevent a possible zero division exception -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
