[
https://issues.apache.org/jira/browse/CASSANDRA-16387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277464#comment-17277464
]
Caleb Rackliffe commented on CASSANDRA-16387:
---------------------------------------------
Nominally, this looks like it's happening is when:
1.) A cluster is created at 3.0.
2.) The schema is updated, in this case a keyspace is created in {{init()}}
and another keyspace and table immediately after, as laid out in {{setup()}}.
3.) The schema migration coordinator node pushes schema mutations out.
4.) Some other node (again still on 3.0) receives the update
message/partitions.
5.) When it tries to gossip its new schema version, it can't even find its
*own endpoint state* in {{endpointStateMap}} in
{{Gossiper#addLocalApplicationStateInternal()}}.
{noformat}
org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions
were thrown during test
at
org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:866)
at
org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:852)
at
org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:188)
at
org.apache.cassandra.distributed.upgrade.MixedModeBatchTestBase.testSimpleStrategy(MixedModeBatchTestBase.java:70)
at
org.apache.cassandra.distributed.upgrade.MixedModeFrom3UnloggedBatchTest.testSimpleStrategy30to4(MixedModeFrom3UnloggedBatchTest.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Suppressed: java.lang.AssertionError
at
org.apache.cassandra.gms.Gossiper.addLocalApplicationStateInternal(Gossiper.java:1551)
at
org.apache.cassandra.gms.Gossiper.addLocalApplicationStates(Gossiper.java:1575)
at
org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1565)
at
org.apache.cassandra.service.MigrationManager.passiveAnnounce(MigrationManager.java:478)
at
org.apache.cassandra.config.Schema.updateVersionAndAnnounce(Schema.java:600)
at
org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1336)
at
org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
at java.lang.Thread.run(Thread.java:748)
{noformat}
This only happens sporadically, so it feels like we're randomly losing the race
to update {{endpointStateMap}}...
> UpgradeTest sporadically failing on schema updates
> --------------------------------------------------
>
> Key: CASSANDRA-16387
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16387
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/java
> Reporter: Caleb Rackliffe
> Assignee: Caleb Rackliffe
> Priority: Normal
> Fix For: 4.0-rc
>
>
> We’ve observed {{UpdateTest}} failing during what appears to be a schema
> change:
> https://app.circleci.com/pipelines/github/maedhroz/cassandra/192/workflows/ed5305e6-e4f9-420e-9f0a-6153333746dc/jobs/1068
> It almost looks like the Gossiper can’t find its own endpoint state in the
> endpoint state map, and the failure is not consistent, which might suggest a
> race.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
