Caleb Rackliffe created CASSANDRA-16495:
-------------------------------------------
Summary: Scheduled (Delayed) Schema Pull Tasks May Run After
MIGRATION Stage Shutdown During Decommission
Key: CASSANDRA-16495
URL: https://issues.apache.org/jira/browse/CASSANDRA-16495
Project: Cassandra
Issue Type: Bug
Components: Local/Startup and Shutdown
Reporter: Caleb Rackliffe
A new test added in CASSANDRA-16181 stumbled across this, although it doesn’t
happen consistently. When [failure
occurs|https://app.circleci.com/pipelines/github/maedhroz/cassandra/235/workflows/eb8133ce-9373-4136-b404-ceca167353f6/jobs/1355/tests],
it appears to be because a delayed schema pull happens after decommission
shuts down the MIGRATION stage’s thread pool.
{noformat}
ERROR [node1_isolatedExecutor:1] node1 2021-02-15 19:35:36,284
CassandraDaemon.java:579 - Exception in thread
Thread[node1_NonPeriodicTasks:1,5,node1]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:72)
at
java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
at
java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:176)
at
java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at org.apache.cassandra.concurrent.Stage.submit(Stage.java:129)
at
org.apache.cassandra.schema.MigrationCoordinator.lambda$scheduleSchemaPull$2(MigrationCoordinator.java:362)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
{noformat}
A fix might be as simple as shutting down ScheduledExecutors.nonPeriodicTasks
in StorageService#decommission(). See the original discussion
[here|https://issues.apache.org/jira/browse/CASSANDRA-16181?focusedCommentId=17293329&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17293329].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
