agagliardi opened a new pull request, #24124:
URL: https://github.com/apache/kafka/pull/24124

   ### Summary
   
   `MirrorCheckpointTask.start()` hands the loading of `CheckpointStore` and 
`OffsetSyncStore` to a `Scheduler` thread so that `start()` does not block 
(KAFKA-17242). `stop()` then closed both stores before it closed the 
`Scheduler`, so a task stopped during a rebalance had its task thread close the 
`KafkaConsumer` that the Scheduler thread was still using inside 
`KafkaBasedLog.start()`. `KafkaConsumer` is not safe for multi-threaded access, 
so whichever thread acquired it second threw `ConcurrentModificationException`; 
when that happened inside `Utils.closeQuietly(consumer, ...)` the consumer was 
also left open.
   
   `KafkaBasedLog.stop()` does not protect against this. It joins its 
`WorkThread`, but that thread is only created after the blocking part of 
`start()`, so while `start()` is still running there is nothing to join and 
`stop()` goes straight on to closing the consumer.
   
   ### Changes
   
   `stop()` now shuts the `Scheduler` down before it closes the stores, so a 
store is only ever closed once the thread that was loading it has finished. To 
keep that wait short, `stop()` first asks each store to stop initializing. A 
store does that by waking up the consumer it owns — the one consumer method 
that is safe to call from another thread — so a load in progress gives up with 
a `WakeupException` instead of reading to the end of the internal topic, and a 
load that has not begun does not start at all. The `WakeupException` caused by 
the shutdown is logged at `DEBUG` instead of being reported as a failure. 
`start()` remains non-blocking.
   
   Closing the `Scheduler` first also keeps the source and target admin clients 
open until the scheduled tasks that use them have finished.
   
   ### Known limitation
   
   `Scheduler.close()`'s `awaitTermination` timeout remains. If it ever 
expires, `close()` logs and returns, `stop()` proceeds, and the race window 
reopens as before. With the wakeup that should require a consumer wedged in 
something `wakeup()` cannot interrupt. This is pre-existing behaviour, shared 
with the other MM2 tasks, and the patch deliberately does not change 
`Scheduler`'s semantics. So: *fixed except for a pre-existing escape hatch*, 
not *proven impossible*.
   
   ### Rejected alternatives
   
   - **Guard the stores with a shutdown flag / make `close()` safe.** A flag 
gives no happens-before edge, and the hazard is two threads inside 
`KafkaConsumer` at the same instant. Doing it properly means holding a lock 
across the whole of `KafkaBasedLog.start()`, which reinstates the unbounded 
wait with extra machinery.
   - **Fix `KafkaBasedLog` so `stop()` waits for an in-progress `start()`.** 
The most general fix, and it would protect every `KafkaBasedLog` user — but 
`KafkaBasedLog` is shared with the Connect herder, where `start()`/`stop()` are 
single-threaded, so this is risk with no local benefit, and it would still 
block `stop()` for the initial read since `KafkaBasedLog` has no "give up 
early" notion. Worth a separate hardening JIRA.
   - **Revert to the blocking `scheduler.execute(...)`.** Reintroduces 
KAFKA-17242's spurious timeout and makes `start()` block again.
   - **`shutdownNow()` / interrupting the scheduler thread.** Turns the blocked 
`poll()` into an `InterruptException` and leaves the consumer in an undefined 
state; `wakeup()` is the supported mechanism and leaves it closeable. 
`Scheduler.close()` is also shared with the other MM2 tasks.
   
   ### Testing
   
   **New tests**:
   - 
`MirrorCheckpointTaskTest.testStopDoesNotCloseOffsetSyncStoreWhileItIsStarting` 
and `...testStopDoesNotCloseCheckpointStoreWhileItIsStarting` block a store 
mid-startup on the scheduler thread, call `stop()` from the task thread, and 
assert the store was not closed while that startup was in progress. Both fail 
deterministically with the current ordering and pass with the fix. 
`CountDownLatch` only, no `Thread.sleep`; the latch is counted down by both the 
fake's `stopInitialization()` and its `close()`, so a wrongly-ordered `stop()` 
fails the assertion rather than hanging.
   - New `OffsetSyncStoreTest` / `CheckpointStoreTest` cases cover 
`stopInitialization()` preventing a start, and the `WakeupException` being 
swallowed only while stopping.
   
   The shipped race tests assert the *ordering invariant* using fakes, which is 
what makes them deterministic; the link from "ordering violated" to 
"`ConcurrentModificationException`" is established separately by the production 
stack traces shown in the JIRA and by a single-threaded-access-checking 
consumer driven through a real `KafkaBasedLog` (verified against unmodified 
`trunk`).
   
   **Test suite results**:
   - `:connect:mirror:test` passes (258 tests, including the MM2 integration 
tests)
   - `:connect:runtime:test --tests '*KafkaBasedLogTest*'` passes
   - `checkstyle` and `spotbugs` pass
   
   ### JIRA
   
   https://issues.apache.org/jira/browse/KAFKA-21435 — a standalone 
container-based
   reproducer is attached to the issue; it runs against the stock
   `apache/kafka:4.1.0` image and needs only podman or docker.
   
   ### Contributor statement
   
   This contribution is my original work and I license the work to the project 
under the project's open source license.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   Co-Authored-By: Claude Opus 5 <[email protected]>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to