[
https://issues.apache.org/jira/browse/CASSANDRA-16495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315055#comment-17315055
]
Ekaterina Dimitrova edited comment on CASSANDRA-16495 at 4/5/21, 7:44 PM:
--------------------------------------------------------------------------
Thank you both!
I just squashed the commits; pushed patches and submitted CI runs for 3.0, 3.11
and 4.0. The only difference is that Stage class was refactored in 4.0 but this
doesn't affect a lot the patch.
[3.0
patch|https://github.com/ekaterinadimitrova2/cassandra/commit/71d06d9b45d907687cef7e1cc6d6210f1b2079c9]
| [CI run |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/583/]
[3.11
patch|https://github.com/ekaterinadimitrova2/cassandra/commit/74a6cbd82bfd3c7177f4067bf63e96114dc0ba85]
| [CI run |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/584/]
[trunk
patch|https://github.com/ekaterinadimitrova2/cassandra/pull/97/commits/ad531e827d4dd88e7d688e79aaa2045c9bb49eb7]
| [CI run |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/585/]
was (Author: e.dimitrova):
Thank you both!
I just squashed the commits and pushed patches and submitted CI runs for 3.0,
3.11 and 4.0. The only difference is that Stage class was refactored in 4.0 but
this doesn't affect a lot the patch.
[3.0 patch|
https://github.com/ekaterinadimitrova2/cassandra/commit/71d06d9b45d907687cef7e1cc6d6210f1b2079c9]
| [CI run | https://jenkins-cm4.apache.org/job/Cassandra-devbranch/583/]
[3.11 patch|
https://github.com/ekaterinadimitrova2/cassandra/commit/74a6cbd82bfd3c7177f4067bf63e96114dc0ba85]
| [CI run | https://jenkins-cm4.apache.org/job/Cassandra-devbranch/584/]
[trunk patch|
https://github.com/ekaterinadimitrova2/cassandra/pull/97/commits/ad531e827d4dd88e7d688e79aaa2045c9bb49eb7]
| [CI run | https://jenkins-cm4.apache.org/job/Cassandra-devbranch/585/]
> Scheduled (Delayed) Schema Pull Tasks May Run After MIGRATION Stage Shutdown
> During Decommission
> ------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16495
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16495
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
> Reporter: Caleb Rackliffe
> Assignee: Ekaterina Dimitrova
> Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-rc
>
>
> A new test added in CASSANDRA-16181 stumbled across this, although it doesn’t
> happen consistently. When [failure
> occurs|https://app.circleci.com/pipelines/github/maedhroz/cassandra/235/workflows/eb8133ce-9373-4136-b404-ceca167353f6/jobs/1355/tests],
> it appears to be because a delayed schema pull happens after decommission
> shuts down the MIGRATION stage’s thread pool.
> {noformat}
> ERROR [node1_isolatedExecutor:1] node1 2021-02-15 19:35:36,284
> CassandraDaemon.java:579 - Exception in thread
> Thread[node1_NonPeriodicTasks:1,5,node1]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:72)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
>
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:176)
>
> at
> java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
> at org.apache.cassandra.concurrent.Stage.submit(Stage.java:129)
> at
> org.apache.cassandra.schema.MigrationCoordinator.lambda$scheduleSchemaPull$2(MigrationCoordinator.java:362)
>
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
> A fix might be as simple as shutting down ScheduledExecutors.nonPeriodicTasks
> in StorageService#decommission(). See the original discussion
> [here|https://issues.apache.org/jira/browse/CASSANDRA-16181?focusedCommentId=17293329&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17293329].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
