SouquieresAdam opened a new pull request, #22917: URL: https://github.com/apache/kafka/pull/22917
## Summary JIRA: [KAFKA-20829](https://issues.apache.org/jira/browse/KAFKA-20829) When a Kafka Streams application requests an app-wide shutdown (e.g. a `StreamsUncaughtExceptionHandler` returning `SHUTDOWN_APPLICATION`), every `StreamThread` on the **classic rebalance protocol** floods the log with: ``` [Consumer instanceId=..., clientId=...-StreamThread-1-consumer, groupId=...] Request joining group due to: Shutdown requested ``` once per run-loop iteration, effectively at CPU speed. ## Root cause `StreamThread.maybeSendShutdown()` is called on every run-loop pass. [KAFKA-19054](https://issues.apache.org/jira/browse/KAFKA-19054) throttled the `log.warn("Detected that shutdown was requested ...")` line to once per 10s, but left the rebalance enforcement unthrottled: ```java if (streamsRebalanceData.isEmpty()) { mainConsumer.enforceRebalance("Shutdown requested"); } ``` `enforceRebalance` reaches the unguarded `AbstractCoordinator.requestRejoin`, which logs `info("Request joining group due to: {}")` unconditionally (no dedup, even when `rejoinNeeded` is already true). During the shutdown rebalance the thread polls non-blocking (`pollPhase()` uses `Duration.ZERO`), so the loop free-spins and the INFO line is emitted every iteration. Only the classic protocol is affected — under the Streams group protocol (KIP-1071) shutdown is signalled through the group heartbeat and never routes through `AbstractCoordinator.requestRejoin`. ## Change Gate the `enforceRebalance` call behind the same 10s throttle that already guards the warning, so both fire together at most once per 10s. The first detection still fires immediately (`lastShutdownWarningTimestamp` starts at `0`), so shutdown is not delayed, and the periodic re-assertion remains as a retry safety net. The fix is confined to the Streams call site; the shared `AbstractCoordinator.requestRejoin` / `ClassicKafkaConsumer.enforceRebalance` are left untouched. ## Testing Added `StreamThreadTest.shouldThrottleEnforceRebalanceOnRepeatedShutdownRequestsUnderClassicProtocol`, which asserts `enforceRebalance("Shutdown requested")` is invoked exactly once across many `maybeSendShutdown()` calls within a 10s window, and once more after the window elapses. Existing regression tests still pass: - `shouldEnforceRebalanceOnShutdownRequestUnderClassicProtocol` (first call still fires immediately) - `shouldNotEnforceRebalanceOnShutdownRequestUnderStreamsProtocol` (KIP-1071 guard preserved) - `shouldNotEnforceRebalanceWhenCurrentlyRebalancing` ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
