[
https://issues.apache.org/jira/browse/CASSANDRA-16495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309478#comment-17309478
]
Caleb Rackliffe edited comment on CASSANDRA-16495 at 3/26/21, 3:33 PM:
-----------------------------------------------------------------------
As long as we agree on the analysis, better stability of
{{HintedHandoffAddRemoveTest}} over time here will be good enough proof for me.
I'll review shortly. In terms of 3.0 and 3.11, I'd favor fixing it there as
well, assuming it merges cleanly (and given how small the change is).
was (Author: maedhroz):
As long as we agree on the analysis, better stability of
{{HintedHandoffAddRemoveTest}} over time here will be good enough proof for me.
I'll review shortly. In terms of 3.0 and 3.11, I'd favor fixing it there as
well, assuming it merges cleanly (and given how small it is).
> Scheduled (Delayed) Schema Pull Tasks May Run After MIGRATION Stage Shutdown
> During Decommission
> ------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16495
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16495
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
> Reporter: Caleb Rackliffe
> Assignee: Ekaterina Dimitrova
> Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-rc
>
>
> A new test added in CASSANDRA-16181 stumbled across this, although it doesn’t
> happen consistently. When [failure
> occurs|https://app.circleci.com/pipelines/github/maedhroz/cassandra/235/workflows/eb8133ce-9373-4136-b404-ceca167353f6/jobs/1355/tests],
> it appears to be because a delayed schema pull happens after decommission
> shuts down the MIGRATION stage’s thread pool.
> {noformat}
> ERROR [node1_isolatedExecutor:1] node1 2021-02-15 19:35:36,284
> CassandraDaemon.java:579 - Exception in thread
> Thread[node1_NonPeriodicTasks:1,5,node1]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:72)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
>
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:176)
>
> at
> java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
> at org.apache.cassandra.concurrent.Stage.submit(Stage.java:129)
> at
> org.apache.cassandra.schema.MigrationCoordinator.lambda$scheduleSchemaPull$2(MigrationCoordinator.java:362)
>
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
> A fix might be as simple as shutting down ScheduledExecutors.nonPeriodicTasks
> in StorageService#decommission(). See the original discussion
> [here|https://issues.apache.org/jira/browse/CASSANDRA-16181?focusedCommentId=17293329&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17293329].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
