Andres de la Peña created CASSANDRA-17674:
---------------------------------------------
Summary: Test failure:
org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage
Key: CASSANDRA-17674
URL: https://issues.apache.org/jira/browse/CASSANDRA-17674
Project: Cassandra
Issue Type: Bug
Components: Test/dtest/java
Reporter: Andres de la Peña
The Java upgrade dtest
{{org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage}}
is ~68% flaky on 4.0 and ~2% flaky on trunk, at least in CircleCI:
* 4.0:
[https://app.circleci.com/pipelines/github/adelapena/cassandra/1622/workflows/0086c3b1-a552-4c7a-8278-2f759cee5bdf/jobs/17288]
* trunk:
[https://app.circleci.com/pipelines/github/adelapena/cassandra/1624/workflows/c4ce2b95-998f-459b-830e-8e3fa6637e15/jobs/17293]
The error for 4.0 is:
{code:java}
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.util.concurrent.TimeoutException:
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting
down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks =
218] did not terminate on time
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at
org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:227)
at
org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.util.concurrent.TimeoutException:
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting
down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks =
218] did not terminate on time
at
org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:235)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.util.concurrent.TimeoutException:
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting
down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks =
218] did not terminate on time
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:231)
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException:
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting
down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks =
218] did not terminate on time
at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:62)
at
org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$28(Instance.java:810)
at
org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$null$8(IsolatedExecutor.java:114)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Caused by: java.util.concurrent.TimeoutException:
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting
down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks =
218] did not terminate on time
at
org.apache.cassandra.utils.ExecutorUtils.awaitTerminationUntil(ExecutorUtils.java:107)
at
org.apache.cassandra.utils.ExecutorUtils.awaitTermination(ExecutorUtils.java:96)
at
org.apache.cassandra.utils.ExecutorUtils.shutdownNowAndWait(ExecutorUtils.java:139)
at
org.apache.cassandra.concurrent.StageManager.shutdownAndWait(StageManager.java:128)
at
org.apache.cassandra.distributed.impl.Instance.lambda$null$26(Instance.java:803)
at
org.apache.cassandra.distributed.impl.Instance.lambda$parallelRun$32(Instance.java:968)
{code}
Whereas the error for trunk is:
{code:java}
org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions
were thrown during test
at
org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:1056)
at
org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1042)
at
org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:244)
at
org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
Suppressed: java.lang.NullPointerException
at
org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:992)
at
org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:987)
at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2512)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:2086)
at
org.apache.cassandra.gms.Gossiper.markAsShutdown(Gossiper.java:479)
at
org.apache.cassandra.gms.GossipShutdownVerbHandler.doVerb(GossipShutdownVerbHandler.java:37)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:69)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
at java.lang.Thread.run(Thread.java:748)
{code}
I haven't managed to reproduce it on 4.1, although more iterations could do it.
This was discovered during CASSANDRA-17213.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
