[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026896#comment-13026896 ] Sylvain Lebresne commented on CASSANDRA-2536: - I'll hijack this conversation by saying that I think we should start advertising that people should try to keep their server clocks in sync unless they have a good reason not too (which would legitimize the fact that timestamp comparison is good enough). Counter removes for instance use server side timestamps and would be screwed up by diverging clocks (and by that I mean more screwed up than they already are by design). And really, is there any reason not to install a ntpd server in the first place anyway? Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Assignee: Tyler Hobbs Attachments: 2536-compare-timestamp.txt, schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026953#comment-13026953 ] Gary Dusbabek commented on CASSANDRA-2536: -- I think timestamp comparisons will be fine. Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Assignee: Tyler Hobbs Attachments: 2536-compare-timestamp.txt, schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026292#comment-13026292 ] Gary Dusbabek commented on CASSANDRA-2536: -- bq. any thoughts... I was going to add some jmx to get the last N schema versions (seems like it would be handy anyway and will be necessary if we ever get the rollback pony). Send schema to node A, verify that schema is propagated to B, send schema to B and watch the problem happen. The code to start looking at are the Definitions*VerbHandlers. Schema version is tracked in two places: gossip and in DatabaseDescriptor.defsVersion. Make sure those are reasonably in sync (was the sourced of one bug in the past). Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Assignee: Gary Dusbabek Attachments: schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026684#comment-13026684 ] Gary Dusbabek commented on CASSANDRA-2536: -- bq. Sometimes the v1 UUID generated by the second node has an earlier timestamp than the current schema UUID has. Wouldn't that update be DOA then? I thought we checked to make sure the new migration compared after the current migration (as well as making sure the new migration's previous version matches with the current version). bq. A node shouldn't accept a schema change if the timestamp for the new schema would be earlier than its current schema. If the clocks are *that* far off sync, I think the cluster has bigger problems (like writes not being applied). Plus, it would be easy for a node whose clock is way head to 'poison' schema updates from the rest of the cluster who are, in effect, behind the times. bq. Schema modification calls should accept an optional client-side timestamp that will be used for the v1 UUID. Seems like a better approach. Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Assignee: Tyler Hobbs Attachments: schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026698#comment-13026698 ] Jonathan Ellis commented on CASSANDRA-2536: --- bq. A node shouldn't accept a schema change if the timestamp for the new schema would be earlier than its current schema. You need this with or without the client-side timestamp, though; there's no sense in letting people blow their leg off. And once you have that you don't need to add a client-side timestamp with all the PITA-ness that involves. (And unlike with data modification, I can't think of a use for doing clever things w/ a client side timesamp. So pushing it to the client doesn't really solve anything, just means you need to sync clocks across more machines.) Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Assignee: Tyler Hobbs Attachments: schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[
https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026710#comment-13026710
]
Tyler Hobbs commented on CASSANDRA-2536:
{quote}
Wouldn't that update be DOA then? I thought we checked to make sure the new
migration compared after the current migration (as well as making sure the new
migration's previous version matches with the current version).
{quote}
We do check that the previous version matches, but the migration is applied
locally without comparing the current and new uuids.
{quote}
If the clocks are that far off sync, I think the cluster has bigger problems
(like writes not being applied).
{quote}
This can theoretically happen with clocks being off by only tens of
milliseconds.
{quote}
And unlike with data modification, I can't think of a use for doing clever
things w/ a client side timesamp. So pushing it to the client doesn't really
solve anything, just means you need to sync clocks across more machines.
{quote}
Not for clever purposes -- it seems to me that clients making schema
modifications are more likely to be centralized, so schema changes coming from
a single client will (almost) always have increasing timestamps.
Schema disagreements when using connections to multiple hosts
-
Key: CASSANDRA-2536
URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8 beta 1
Environment: Two node 0.8-beta1 cluster with one seed and JNA.
Reporter: Tyler Hobbs
Assignee: Tyler Hobbs
Attachments: schema_disagree.py
If you have two thrift connections open to different nodes and you create a
KS using the first, then a CF in that KS using the second, you wind up with a
schema disagreement even if you wait/sleep after creating the KS.
The attached script reproduces the issue using pycassa (1.0.6 should work
fine, although it has the 0.7 thrift-gen code). It's also reproducible by
hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026070#comment-13026070 ] Jonathan Ellis commented on CASSANDRA-2536: --- Gary, any thoughts on where to start looking? Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Assignee: Gary Dusbabek Attachments: schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13024804#comment-13024804 ] Brian Lovett commented on CASSANDRA-2536: - Just bumped into this on a fresh 0.7.4 install on our test cluster. Does this only happen in a 2 node ring? Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Assignee: Gary Dusbabek Attachments: schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023257#comment-13023257 ] Tyler Hobbs commented on CASSANDRA-2536: Actually, I can also reproduce this with a two node 0.7.4 cluster. I'm pretty sure that this does not happen with 0.7.3, but I'll go ahead and verify that. Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Assignee: Gary Dusbabek Attachments: schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023263#comment-13023263 ] Tyler Hobbs commented on CASSANDRA-2536: Nevermind my thoughts that it doesn't happen in 0.7.3 -- it seems to happen there too. It appears this is not a recent problem. Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Assignee: Gary Dusbabek Attachments: schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using connections to multiple hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022978#comment-13022978 ] Mike Bulman commented on CASSANDRA-2536: I feel like a better, more critical sounding explanation, is: create a keyspace on node1, create a cf in that keyspace on node2 = hang + schema disagreement. Schema disagreements when using connections to multiple hosts - Key: CASSANDRA-2536 URL: https://issues.apache.org/jira/browse/CASSANDRA-2536 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Environment: Two node 0.8-beta1 cluster with one seed and JNA. Reporter: Tyler Hobbs Attachments: schema_disagree.py If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS. The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
