Michael Semb Wever created CASSANDRA-21696:
----------------------------------------------
Summary: Request callbacks registered after messaging service
shutdown are never expired
Key: CASSANDRA-21696
URL: https://issues.apache.org/jira/browse/CASSANDRA-21696
Project: Apache Cassandra
Issue Type: Bug
Components: Messaging/Internode
Reporter: Michael Semb Wever
{{RequestCallbacks.addWithExpiration}} registers a callback whatever the state
of the reaper executor. A request that reaches the coordinator after
{{MessagingService.shutdown}} has called {{shutdownNow}} leaves an entry that
nothing expires, because the reaper is already shut down, so the entry holds
its message and its response handler for the life of the process.
CASSANDRA-6948 repaired the same collection of callbacks for the 2.0 expiring
map, when the reaper could not be restarted after bootstrap or replace.
The graceful path is worse, because it reschedules itself on the same executor
while the map is not empty:
{code:java}
void shutdownGracefully()
{
expire();
if (!callbacks.isEmpty())
executor.schedule(this::shutdownGracefully, 100L, MILLISECONDS);
else
executor.shutdownNow();
}
{code}
A steady arrival of new requests keeps the map populated, so the reaper is
never shut down, {{awaitTerminationUntil}} throws {{TimeoutException}}, and
{{MessagingService.shutdown}} waits out its whole deadline before it reports
the timeout.
The patch sets a flag in both shutdown methods, and each {{addWithExpiration}}
overload then reports the late request at trace level and registers nothing. A
caller is unaffected: the message is not sent either, so the caller ends on its
own deadline, as it does today for a callback the reaper expires.
Patch:
[mck/upstream/callbacks-after-shutdown/5.0|https://github.com/thelastpickle/cassandra/tree/mck/upstream/callbacks-after-shutdown/5.0]
Provenance:
[8aa35c6cab|https://github.com/datastax/cassandra/commit/8aa35c6cab8ccd0ce4326fc60f50c2f9c02e4108]
by [~mike_tr_adamson]. That commit carries the same flag and guards; this
patch names the message id and the peer in the trace line and adds the
regression test the commit lacks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
