[
https://issues.apache.org/jira/browse/CASSANDRA-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17334402#comment-17334402
]
David Capwell commented on CASSANDRA-16238:
-------------------------------------------
If this is a race reading un-committed data, then the patch below might work
around it (double check locking, but with queues)
Patch:
{code}
$ git diff
diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java
b/src/java/org/apache/cassandra/gms/Gossiper.java
index 699f235bd3..e82b107bac 100644
--- a/src/java/org/apache/cassandra/gms/Gossiper.java
+++ b/src/java/org/apache/cassandra/gms/Gossiper.java
@@ -928,6 +928,11 @@ public class Gossiper implements
IFailureDetectionEventListener, GossiperMBean
{
logger.info("FatClient {} has been silent for {}ms,
removing from gossip", endpoint, fatClientTimeout);
runInGossipStageBlocking(() -> {
+ if (!isGossipOnlyMember(endpoint))
+ {
+ // updating gossip and token metadata are not
atomic, but rely on the single threaded gossip stage
+ // since status checks are done outside the gossip
stage, need to confirm the state of the endpoint
+ // to make sure that the previous read data was
correct
+ logger.info("Race condition marking {} as a
FatClient; ignoring", endpoint);
+ return;
+ }
removeEndpoint(endpoint); // will put it in
justRemovedEndpoints to respect quarantine delay
evictFromMembership(endpoint); // can get rid of the
state immediately
});
{code}
> Fix flaky jvm-dtests that fail with Unable to contact any seeds
> ---------------------------------------------------------------
>
> Key: CASSANDRA-16238
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16238
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/java
> Reporter: David Capwell
> Assignee: Brandon Williams
> Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: 16238-archived-failures.txt
>
>
> https://app.circleci.com/pipelines/github/dcapwell/cassandra/745/workflows/1c7e589e-b5af-4a56-b40a-43da424602c7/jobs/4231
> {code}
> test teardown failure
> Unexpected error found in node logs (see stdout for full details). Errors:
> [ERROR [main] 2020-10-29 17:38:13,808 CassandraDaemon.java:817 - Exception
> encountered during startup
> java.lang.IllegalStateException: Unable to contact any seeds!
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1601)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:699)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:635)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795),
> ERROR [main] 2020-10-29 17:38:13,808 CassandraDaemon.java:817 - Exception
> encountered during startup
> java.lang.IllegalStateException: Unable to contact any seeds!
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1601)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:699)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:635)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
