[
https://issues.apache.org/jira/browse/CASSANDRA-16495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315861#comment-17315861
]
Ekaterina Dimitrova edited comment on CASSANDRA-16495 at 4/6/21, 10:04 PM:
---------------------------------------------------------------------------
j8_dtests_jars_build fails on all branches plus on latest trunk both locally
and in CircleCI so it is not related to this patch. I will try to investigate
it separately.
trunk: [JAVA 8
CI|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/743/workflows/0fb184bb-1852-4ef5-abb9-bb52bae17091]
| [JAVA 11
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/743/workflows/35cea1fd-6f38-4f09-9d7f-4673c34a9851]
_test_describe - cqlsh_tests.test_cqlsh.TestCqlsh_ - failing in both Java11;
seems unrelated?
_org.apache.cassandra.net.AsyncPromiseTest_ - failing in both Java11; seems
unrelated?
3.11: [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/741/workflows/c3de5483-85b0-48a4-b1fa-1c52f89680c7]
_test_multiple_repair - repair_tests.incremental_repair_test.TestIncRepair_ -
-known failure;- no failures in Jenkins, I was mistaken but seems unrelated
timeout
_test_closing_connections - thrift_hsha_test.TestThriftHSHA - I think it is
known failure; I've seen it regularly in Circle_
3.0: [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/740/workflows/02ad5584-60de-48c3-ba11-565732ac3dea]
_test_closing_connections - thrift_hsha_test.TestThriftHSHA_ - I think it is
known failure
I see also the following warning which needs to be fixed separately and it is
unrelated to this patch:
{code:java}
WARNING: multiple versions of ant detected in path for junit
jar:file:/usr/share/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
and
jar:file:/tmp/cassandra/build/test/lib/jars/ant-1.9.7.jar!/org/apache/tools/ant/Project.class
{code}
was (Author: e.dimitrova):
j8_dtests_jars_build fails on all branches plus on latest trunk both locally
and in CircleCI so it is not related to this patch. I will try to investigate
it separately.
trunk: [JAVA 8
CI|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/743/workflows/0fb184bb-1852-4ef5-abb9-bb52bae17091]
| [JAVA 11
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/743/workflows/35cea1fd-6f38-4f09-9d7f-4673c34a9851]
_test_describe - cqlsh_tests.test_cqlsh.TestCqlsh_ - failing in both Java11;
seems unrelated?
_org.apache.cassandra.net.AsyncPromiseTest_ - failing in both Java11; seems
unrelated?
3.11: [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/741/workflows/c3de5483-85b0-48a4-b1fa-1c52f89680c7]
_test_multiple_repair - repair_tests.incremental_repair_test.TestIncRepair_ -
known failure
_test_closing_connections - thrift_hsha_test.TestThriftHSHA - I think it is
known failure_
3.0: [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/740/workflows/02ad5584-60de-48c3-ba11-565732ac3dea]
_test_closing_connections - thrift_hsha_test.TestThriftHSHA_ - I think it is
known failure
I see also the following warning which needs to be fixed separately and it is
unrelated to this patch:
{code:java}
WARNING: multiple versions of ant detected in path for junit
jar:file:/usr/share/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
and
jar:file:/tmp/cassandra/build/test/lib/jars/ant-1.9.7.jar!/org/apache/tools/ant/Project.class
{code}
> Scheduled (Delayed) Schema Pull Tasks May Run After MIGRATION Stage Shutdown
> During Decommission
> ------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16495
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16495
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
> Reporter: Caleb Rackliffe
> Assignee: Ekaterina Dimitrova
> Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-rc
>
>
> A new test added in CASSANDRA-16181 stumbled across this, although it doesn’t
> happen consistently. When [failure
> occurs|https://app.circleci.com/pipelines/github/maedhroz/cassandra/235/workflows/eb8133ce-9373-4136-b404-ceca167353f6/jobs/1355/tests],
> it appears to be because a delayed schema pull happens after decommission
> shuts down the MIGRATION stage’s thread pool.
> {noformat}
> ERROR [node1_isolatedExecutor:1] node1 2021-02-15 19:35:36,284
> CassandraDaemon.java:579 - Exception in thread
> Thread[node1_NonPeriodicTasks:1,5,node1]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:72)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
>
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:176)
>
> at
> java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
> at org.apache.cassandra.concurrent.Stage.submit(Stage.java:129)
> at
> org.apache.cassandra.schema.MigrationCoordinator.lambda$scheduleSchemaPull$2(MigrationCoordinator.java:362)
>
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
> A fix might be as simple as shutting down ScheduledExecutors.nonPeriodicTasks
> in StorageService#decommission(). See the original discussion
> [here|https://issues.apache.org/jira/browse/CASSANDRA-16181?focusedCommentId=17293329&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17293329].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
