Besides the obviously confusing error message, this particular case could simply be that the hash value of the primary key belonged to the other node that wasn’t up, so even though one node was up, it didn’t own that particular hash value or token, so CL=ONE could not succeed.
What was RF set to for this two node cluster? -- Jack Krupansky From: Andrew Sent: Wednesday, July 23, 2014 1:02 AM To: graham sanderson ; user@cassandra.apache.org Cc: Kevin Burton Subject: Re: All writes fail with ONE consistency level when adding second node to cluster? I looked into this; ONE means it must be written to one replica—i.e., a node the data is supposed to be written to. ANY means a hinted handoff will “count”. So as long as it writes to any node on the cluster—even one that it’s not supposed to be on—it will be a success. Good to know. Andrew On July 22, 2014 at 8:13:57 PM, graham sanderson (gra...@vast.com) wrote: Incorrect, ONE does not refer to the number of “other" nodes, it just refers to the number of nodes. so ONE under normal circumstances would only require one node to acknowledge the write. The confusing error message you are getting is related to https://issues.apache.org/jira/browse/CASSANDRA-833… Kevin you are correct in that normally that error message would make no sense. I don’t have much experience adding/removing nodes, but I think what is happening is that your new node is in the middle of taken over ownership of a token range - while that happens C* is trying to write to both the old owner (your original node), AND (hence the 2 not 1 in the error message) the new owner (the new node) so that once the bootstrapping of the new node is complete, it is immediately safe to delete the (no longer owned data) from the old node. For whatever reason the write to the new node is timing out, causing the exception, and the error message is exposing the “2” which happens to be how many C* thinks it is waiting for at the time (i.e. how many it should be waiting for based on the consistency level (1) plus this extra node). On Jul 22, 2014, at 9:46 PM, Andrew <redmu...@gmail.com> wrote: ONE means write to one replica (in addition to the original). If you want to write to any of them, use ANY. Is that the right understanding? http://www.datastax.com/docs/1.0/dml/data_consistency Andrew On July 22, 2014 at 7:43:43 PM, Kevin Burton (bur...@spinn3r.com) wrote: I'm super confused by this.. and disturbed that this was my failure scenario :-( I had one cassandra node for the alpha of my app… and now we're moving into beta… which means three replicas. So I added the second node… but my app immediately broke with: ""Cassandra timeout during write query at consistency ONE (2 replica were required but only 1 acknowledged the write)"" … but that makes no sense… if I'm at ONE and I have one acknowledged write, why does it matter that the second one hasn't ack'd yet… ? -- Founder/CEO Spinn3r.com Location: San Francisco, CA blog: http://burtonator.wordpress.com … or check out my Google+ profile ------------------------------------------------------------------------------
