[
https://issues.apache.org/jira/browse/CASSANDRA-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-21696:
-------------------------------------------
Test and Documentation Plan:
The new RequestCallbacksTest (2 tests) passes on JDK 17, and both tests fail
with the two guards removed and the rest of the patch in place.
testNoCallbackIsRegisteredAfterShutdownNow holds the leak, and
testGracefulShutdownTerminatesWhenARequestArrivesLate holds the reschedule
loop: it leaves one callback outstanding, starts the graceful shutdown, adds a
late request, removes the outstanding callback, and then waits for the reaper
to terminate.
MessagingServiceTest passes apart from four listen tests that need a 127.0.0.2
loopback alias this machine does not have; those four fail the same way without
the patch.
Status: Patch Available (was: Open)
> Request callbacks registered after messaging service shutdown are never
> expired
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-21696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21696
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
> Reporter: Michael Semb Wever
> Priority: Normal
> Fix For: 5.0.x, 6.0.x, 7.x
>
>
> {{RequestCallbacks.addWithExpiration}} registers a callback whatever the
> state of the reaper executor. A request that reaches the coordinator after
> {{MessagingService.shutdown}} has called {{shutdownNow}} leaves an entry that
> nothing expires, because the reaper is already shut down, so the entry holds
> its message and its response handler for the life of the process.
> CASSANDRA-6948 repaired the same collection of callbacks for the 2.0 expiring
> map, when the reaper could not be restarted after bootstrap or replace.
> The graceful path is worse, because it reschedules itself on the same
> executor while the map is not empty:
> {code:java}
> void shutdownGracefully()
> {
> expire();
> if (!callbacks.isEmpty())
> executor.schedule(this::shutdownGracefully, 100L, MILLISECONDS);
> else
> executor.shutdownNow();
> }
> {code}
> A steady arrival of new requests keeps the map populated, so the reaper is
> never shut down, {{awaitTerminationUntil}} throws {{TimeoutException}}, and
> {{MessagingService.shutdown}} waits out its whole deadline before it reports
> the timeout.
> The patch sets a flag in both shutdown methods, and each
> {{addWithExpiration}} overload then reports the late request at trace level
> and registers nothing. A caller is unaffected: the message is not sent
> either, so the caller ends on its own deadline, as it does today for a
> callback the reaper expires.
> Patch:
> [mck/upstream/callbacks-after-shutdown/5.0|https://github.com/thelastpickle/cassandra/tree/mck/upstream/callbacks-after-shutdown/5.0]
> Provenance:
> [8aa35c6cab|https://github.com/datastax/cassandra/commit/8aa35c6cab8ccd0ce4326fc60f50c2f9c02e4108]
> by [~mike_tr_adamson]. That commit carries the same flag and guards; this
> patch names the message id and the peer in the trace line and adds the
> regression test the commit lacks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
