agagliardi opened a new pull request, #24124: URL: https://github.com/apache/kafka/pull/24124
### Summary `MirrorCheckpointTask.start()` hands the loading of `CheckpointStore` and `OffsetSyncStore` to a `Scheduler` thread so that `start()` does not block (KAFKA-17242). `stop()` then closed both stores before it closed the `Scheduler`, so a task stopped during a rebalance had its task thread close the `KafkaConsumer` that the Scheduler thread was still using inside `KafkaBasedLog.start()`. `KafkaConsumer` is not safe for multi-threaded access, so whichever thread acquired it second threw `ConcurrentModificationException`; when that happened inside `Utils.closeQuietly(consumer, ...)` the consumer was also left open. `KafkaBasedLog.stop()` does not protect against this. It joins its `WorkThread`, but that thread is only created after the blocking part of `start()`, so while `start()` is still running there is nothing to join and `stop()` goes straight on to closing the consumer. ### Changes `stop()` now shuts the `Scheduler` down before it closes the stores, so a store is only ever closed once the thread that was loading it has finished. To keep that wait short, `stop()` first asks each store to stop initializing. A store does that by waking up the consumer it owns — the one consumer method that is safe to call from another thread — so a load in progress gives up with a `WakeupException` instead of reading to the end of the internal topic, and a load that has not begun does not start at all. The `WakeupException` caused by the shutdown is logged at `DEBUG` instead of being reported as a failure. `start()` remains non-blocking. Closing the `Scheduler` first also keeps the source and target admin clients open until the scheduled tasks that use them have finished. ### Known limitation `Scheduler.close()`'s `awaitTermination` timeout remains. If it ever expires, `close()` logs and returns, `stop()` proceeds, and the race window reopens as before. With the wakeup that should require a consumer wedged in something `wakeup()` cannot interrupt. This is pre-existing behaviour, shared with the other MM2 tasks, and the patch deliberately does not change `Scheduler`'s semantics. So: *fixed except for a pre-existing escape hatch*, not *proven impossible*. ### Rejected alternatives - **Guard the stores with a shutdown flag / make `close()` safe.** A flag gives no happens-before edge, and the hazard is two threads inside `KafkaConsumer` at the same instant. Doing it properly means holding a lock across the whole of `KafkaBasedLog.start()`, which reinstates the unbounded wait with extra machinery. - **Fix `KafkaBasedLog` so `stop()` waits for an in-progress `start()`.** The most general fix, and it would protect every `KafkaBasedLog` user — but `KafkaBasedLog` is shared with the Connect herder, where `start()`/`stop()` are single-threaded, so this is risk with no local benefit, and it would still block `stop()` for the initial read since `KafkaBasedLog` has no "give up early" notion. Worth a separate hardening JIRA. - **Revert to the blocking `scheduler.execute(...)`.** Reintroduces KAFKA-17242's spurious timeout and makes `start()` block again. - **`shutdownNow()` / interrupting the scheduler thread.** Turns the blocked `poll()` into an `InterruptException` and leaves the consumer in an undefined state; `wakeup()` is the supported mechanism and leaves it closeable. `Scheduler.close()` is also shared with the other MM2 tasks. ### Testing **New tests**: - `MirrorCheckpointTaskTest.testStopDoesNotCloseOffsetSyncStoreWhileItIsStarting` and `...testStopDoesNotCloseCheckpointStoreWhileItIsStarting` block a store mid-startup on the scheduler thread, call `stop()` from the task thread, and assert the store was not closed while that startup was in progress. Both fail deterministically with the current ordering and pass with the fix. `CountDownLatch` only, no `Thread.sleep`; the latch is counted down by both the fake's `stopInitialization()` and its `close()`, so a wrongly-ordered `stop()` fails the assertion rather than hanging. - New `OffsetSyncStoreTest` / `CheckpointStoreTest` cases cover `stopInitialization()` preventing a start, and the `WakeupException` being swallowed only while stopping. The shipped race tests assert the *ordering invariant* using fakes, which is what makes them deterministic; the link from "ordering violated" to "`ConcurrentModificationException`" is established separately by the production stack traces shown in the JIRA and by a single-threaded-access-checking consumer driven through a real `KafkaBasedLog` (verified against unmodified `trunk`). **Test suite results**: - `:connect:mirror:test` passes (258 tests, including the MM2 integration tests) - `:connect:runtime:test --tests '*KafkaBasedLogTest*'` passes - `checkstyle` and `spotbugs` pass ### JIRA https://issues.apache.org/jira/browse/KAFKA-21435 — a standalone container-based reproducer is attached to the issue; it runs against the stock `apache/kafka:4.1.0` image and needs only podman or docker. ### Contributor statement This contribution is my original work and I license the work to the project under the project's open source license. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
