[
https://issues.apache.org/jira/browse/CASSANDRA-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071298#comment-13071298
]
Brandon Williams commented on CASSANDRA-2946:
---------------------------------------------
+1, what happened was node A never updated it's gossip state, but the other
node learned of its schema version by RPC. When node B restarted, it no longer
knew the schema version for A, and tried to get it from gossip where it was
still old.
bq. if it does we can also change the HH code to use DD.
I don't know about that, having this as a check to make sure this doesn't
happen again is a far easier thing to catch than other "gossip is out of date"
problems we might encounter.
> HintedHandoff fails with could not reach schema agreement
> ---------------------------------------------------------
>
> Key: CASSANDRA-2946
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Reporter: Richard Low
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2946.txt
>
>
> To reproduce, have two nodes A and B.
> 1. On node A, create a keyspace with replication factor 1 and add a column
> family
> 2. Ensure node B has created the keyspace and column family
> 3. Take down node B
> 4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
> 5. Bring up node B
> 6. When hints are delivered, I get the error:
> ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java
> (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach
> schema agreement with /10.2.129.9 in 60000ms
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with
> /10.2.129.9 in 60000ms
> at
> org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
> at
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
> at
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> If I use DatabaseDescriptor.getDefsVersion() instead of
> gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA)
> then the error goes away, and the hints are correctly delivered.
> This may be the same issue as Aaron saw here:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html,
> and may be related to CASSANDRA-2083.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
