[
https://issues.apache.org/jira/browse/CASSANDRA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118677#comment-13118677
]
Cathy Daw edited comment on CASSANDRA-3290 at 10/1/11 5:33 AM:
---------------------------------------------------------------
I see the same issue upgrading from 0.8.2, but now I see the following
exception. In this case, one node was drained and upgraded to 1.0 and the other
two nodes were still at 0.8.2.
This is a pretty straight forward procedure using a vanilla install and the out
of the box stress scripts.
{code}
ERROR 05:27:33,751 Fatal exception in thread Thread[HintedHandoff:1,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema
agreement with /50.57.111.189 in 60000ms
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Could not reach schema agreement with
/50.57.111.189 in 60000ms
at
org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:220)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:231)
at
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
at
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:333)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
ERROR 05:27:33,753 Fatal exception in thread Thread[HintedHandoff:1,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema
agreement with /50.57.111.189 in 60000ms
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Could not reach schema agreement with
/50.57.111.189 in 60000ms
at
org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:220)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:231)
at
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
at
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:333)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
{code}
*Node Tool output for 1 node in 1.0.0 and the other two at 0.8.2*
{code}
cathy1:~/apache-cassandra-1.0.0-rc1-src/bin# ./nodetool -h cathy1 ring
Address DC Rack Status State Load Owns
Token
113427455640312821154458202477256070484
50.57.111.111 datacenter1 rack1 Up Normal 181.56 MB 33.33%
0
50.57.111.126 datacenter1 rack1 Up Normal 178.84 MB 33.33%
56713727820156410577229101238628035242
50.57.111.189 datacenter1 rack1 Up Normal 180.2 MB 33.33%
113427455640312821154458202477256070484
{code}
was (Author: cdaw):
I see the same issue upgrading from 0.8.2
> Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on
> data created in 0.7.9
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3290
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Cathy Daw
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
>
> *TEST CASE*
> * Install three node cluster with C* 0.7.9 with node1 as the seed
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o
> insert
> * Run `nodetool -h node1 drain`
> * Upgrade binaries and edit cassandra.yaml for node1.
> * Startup C* 1.0 on node
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o
> insert
> *ISSUE*
> * Run the 0.7.9 stress tool to read data: ./stress -d node2 -o read
> {code}
> cathy3:~/apache-cassandra-0.7.9-src/contrib/stress/bin# ./stress -d cathy3 -o
> read
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> Operation [17] retried 10 times - error reading key 0521977
> Operation [9] retried 10 times - error reading key 0595365
> Operation [15] retried 10 times - error reading key 0513247
> {code}
> *Additional notes*
> * There are no errors in the log files
> * After upgrading the full ring, and running READ from both the old and new
> versions of stress tool, the error still exists
> * Once all nodes are upgraded, if I run INSERT followed by READ, the new
> reads are fine
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
