This is an automated email from the ASF dual-hosted git repository.
aleksey pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 7ca997b Drain and/or shutdown might throw because of slow messaging
service shutdown
7ca997b is described below
commit 7ca997ba3514e19864d53b8ca56a1e4f5c26208f
Author: Aleksey Yeshchenko <[email protected]>
AuthorDate: Tue Oct 20 18:02:45 2020 +0100
Drain and/or shutdown might throw because of slow messaging service shutdown
patch by Aleksey Yeschenko; reviewed by Marcus Eriksson and Alex Petrov
for CASSANDRA-16276
---
CHANGES.txt | 1 +
.../org/apache/cassandra/service/StorageService.java | 18 +++++++++++++++---
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 29100fc..176bc04 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0-beta4
+ * Drain and/or shutdown might throw because of slow messaging service
shutdown (CASSANDRA-16276)
* Upgrade JNA to 5.6.0, dropping support for <=glibc-2.6 systems
(CASSANDRA-16212)
* Add saved Host IDs to TokenMetadata at startup (CASSANDRA-16246)
* Ensure that CacheMetrics.requests is picked up by the metric reporter
(CASSANDRA-16228)
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index 7d27163..9c6499a 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -4746,15 +4746,27 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
if (!isFinalShutdown)
setMode(Mode.DRAINING, "shutting down MessageService", false);
- // In-progress writes originating here could generate hints to be
written, so shut down MessagingService
- // before mutation stage, so we can get all the hints saved before
shutting down
- MessagingService.instance().shutdown();
+ // In-progress writes originating here could generate hints to be
written,
+ // which is currently scheduled on the mutation stage. So shut
down MessagingService
+ // before mutation stage, so we can get all the hints saved before
shutting down.
+ try
+ {
+ MessagingService.instance().shutdown();
+ }
+ catch (Throwable t)
+ {
+ // prevent messaging service timing out shutdown from aborting
+ // drain process; otherwise drain and/or shutdown might throw
+ logger.error("Messaging service timed out shutting down", t);
+ }
if (!isFinalShutdown)
setMode(Mode.DRAINING, "clearing mutation stage", false);
viewMutationStage.shutdown();
counterMutationStage.shutdown();
mutationStage.shutdown();
+
+ // FIXME? should these *really* take up to one hour?
viewMutationStage.awaitTermination(3600, TimeUnit.SECONDS);
counterMutationStage.awaitTermination(3600, TimeUnit.SECONDS);
mutationStage.awaitTermination(3600, TimeUnit.SECONDS);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
