Re: Schema version mismatch

[Nandakishore Tokala]

FYI

this is the info I got after research on schema version

what is the schema version in Cassandra and for what are all the changes
schema version is changed?

schema version is a UUID used to set a baseline version of the schema. It
tracks if a schema migration is required. Schema version changes when there
is any schema changes made to the cluster like adding or dropping
tables/column/keyspaces i.e CREATE, DROP, ALTER etc.

why do we see schema version mismatch other than upgrades time?

It can be seen when schema changes are made i.e DDL statements are issues
concurrently.
If a node is down and schema changes are being made.
NTP is not in sync
If the nodes in the cluster are running with mixed  Apache versions.
Cassandra schema updates assume that schema changes are done one-at-a-time.
If you make multiple changes at the same time, you can cause some nodes to
end up with a different schema, than others.

Can we solve the schema version mismatch without restart the Cassandra nodes

When you only have one node in disagreement, the easiest way is simply to
run nodetool resetlocalschema, which will make that node forget it's schema
and request it again from the other nodes.

When you have more than one node in disagreement, it becomes more
difficult, because you can't control which node will respond to the schema
request, so one of the nodes with the wrong schema could send it back to
the node where you ran resetlocalschema, and then it will still be in
disagreement with the majority.

In this case, you need to shut down all the nodes that have the incorrect
schema, then start them up one node at a time and after each node comes up,
run nodetool resetlocalschema. Check that the schema is now in agreement
for all the nodes that are currently running, and then repeat the process
for each remaining node that has the incorrect schema.

On Thu, Jun 7, 2018 at 4:54 PM, Nandakishore Tokala <
nandakishore.tok...@gmail.com> wrote:

> Thanks Justin
>
> On Thu, Jun 7, 2018 at 4:27 PM, Justin Cameron 
> wrote:
>
>> You may run into schema mismatch issues if you make multiple schema
>> alterations in a very short period of time (e.g. if you're programmatically
>> modifying tables 50x a second). You'll be better off making schema changes
>> in advance. If you need to make dynamic changes then you could check the
>> version to make sure that the previous change has applied before applying
>> the next one.
>>
>> AFAIK the only way to resolve schema mismatch is a rolling restart
>>
>> On Fri, 8 Jun 2018 at 03:03 Nandakishore Tokala <
>> nandakishore.tok...@gmail.com> wrote:
>>
>>> what is the schema version in Cassandra and for what are all the changes
>>> schema version is changed?
>>>
>>> why do we see schema version mismatch other than upgrades time?
>>> Can we solve the schema version mismatch without restart the
>>> Cassandra nodes
>>>
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Nanda Kishore
>>>
>> --
>>
>>
>> *Justin Cameron*Senior Software Engineer
>>
>>
>> <https://www.instaclustr.com/>
>>
>>
>> This email has been sent on behalf of Instaclustr Pty. Limited
>> (Australia) and Instaclustr Inc (USA).
>>
>> This email and any attachments may contain confidential and legally
>> privileged information.  If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>>
>
>
>
> --
> Thanks & Regards,
> Nanda Kishore
>



-- 
Thanks & Regards,
Nanda Kishore

Re: Schema version mismatch

[Nandakishore Tokala]

Thanks Justin

On Thu, Jun 7, 2018 at 4:27 PM, Justin Cameron 
wrote:

> You may run into schema mismatch issues if you make multiple schema
> alterations in a very short period of time (e.g. if you're programmatically
> modifying tables 50x a second). You'll be better off making schema changes
> in advance. If you need to make dynamic changes then you could check the
> version to make sure that the previous change has applied before applying
> the next one.
>
> AFAIK the only way to resolve schema mismatch is a rolling restart
>
> On Fri, 8 Jun 2018 at 03:03 Nandakishore Tokala <
> nandakishore.tok...@gmail.com> wrote:
>
>> what is the schema version in Cassandra and for what are all the changes
>> schema version is changed?
>>
>> why do we see schema version mismatch other than upgrades time?
>> Can we solve the schema version mismatch without restart the
>> Cassandra nodes
>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Nanda Kishore
>>
> --
>
>
> *Justin Cameron*Senior Software Engineer
>
>
> <https://www.instaclustr.com/>
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>



-- 
Thanks & Regards,
Nanda Kishore

Re: Schema version mismatch

[Justin Cameron]

You may run into schema mismatch issues if you make multiple schema
alterations in a very short period of time (e.g. if you're programmatically
modifying tables 50x a second). You'll be better off making schema changes
in advance. If you need to make dynamic changes then you could check the
version to make sure that the previous change has applied before applying
the next one.

AFAIK the only way to resolve schema mismatch is a rolling restart

On Fri, 8 Jun 2018 at 03:03 Nandakishore Tokala <
nandakishore.tok...@gmail.com> wrote:

> what is the schema version in Cassandra and for what are all the changes
> schema version is changed?
>
> why do we see schema version mismatch other than upgrades time?
> Can we solve the schema version mismatch without restart the
> Cassandra nodes
>
>
>
>
> --
> Thanks & Regards,
> Nanda Kishore
>
-- 


*Justin Cameron*Senior Software Engineer


<https://www.instaclustr.com/>


This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Schema version mismatch

[Nandakishore Tokala]

what is the schema version in Cassandra and for what are all the changes
schema version is changed?

why do we see schema version mismatch other than upgrades time?
Can we solve the schema version mismatch without restart the Cassandra nodes



-- 
Thanks & Regards,
Nanda Kishore

Re: Schema version mismatch with 3.0.8 and 3.0.14

[Jon Haddad]

Generally speaking, I would never advise someone to add nodes to a cluster 
using a different version than the rest of the cluster.  

> On Dec 1, 2017, at 11:58 AM, Jai Bheemsen Rao Dhanwada 
>  wrote:
> 
> Thanks Jeff,
> 
> I did some more testing on this version upgrade and here is brief summary 
> incase if someone runs into the same issue.
> 
> Basically, we cannot add new nodes with 3.0.14 or higher version to the 
> cluster with 3.0.8 version of cassandra (haven't tested with 3.0.13) and 
> below are the two options available
> 
> 1. Rolling upgrade all the nodes in the cluster to 3.0.14 and add new nodes 
> with the same version 3.0.14
> OR
> 2. use the flag "-Dcassandra.force_3_0_protocol_version=true" to join the 
> nodes with higher version.
> 
> On Thu, Nov 30, 2017 at 9:45 AM, Jeff Jirsa  > wrote:
> I guess I didn’t realize you were adding and upgrading s
> at the same time
> 
> Do one or the other first (either add the new node at the old version, or 
> upgrade to the new version and then add the new node), or use the two step 
> process described in 13004
> 
> -- 
> Jeff Jirsa
> 
> 
> On Nov 30, 2017, at 9:40 AM, Jai Bheemsen Rao Dhanwada  > wrote:
> 
>> Hello Jeff,
>> 
>> I tried with 3.0.15 as well and I see the same issue.
>> 
>> Here are the steps I followed.
>> 
>> 1. I have a 3 node C* cluster with 3.0.8 version
>> 2. Added a new node (4th) node with 3.0.15 version of cassandra
>> 3. New node joins the cluster but it is creates it's own schema and don't 
>> bootstrap any keyspace or column families from the cluster.
>> 
>> in the logs I could see the below WARN message.
>> 
>> OutboundTcpConnection.java:488 - Seed gossip version is 10; will not connect 
>> with that version
>> 
>> so we cannot join a new node with the higher minor version to the cluster?
>> 
>> On Wed, Nov 29, 2017 at 5:16 PM, Jeff Jirsa > > wrote:
>> Yes, going to 3.0.14 or 3.0.15 should work correctly
>> 
>> 3.0.15 will have important bug fixes not in 3.0.14
>> 
>> 
>> -- 
>> Jeff Jirsa
>> 
>> 
>> On Nov 29, 2017, at 4:52 PM, Jai Bheemsen Rao Dhanwada 
>> > wrote:
>> 
>>> Thanks Jeff,
>>> 
>>> You mean if I upgrade from 3.0.8 -> 3.0.15 it should work without any 
>>> issues?
>>> 
>>> On Wed, Nov 29, 2017 at 4:21 PM, Jeff Jirsa >> > wrote:
>>> This is because in 3.0.14, we had to bump the internode messaging version 
>>> to fix a nasty bug (it's something we almost never do in minor versions, 
>>> but it was necessary here), and schema changes won't cross internode 
>>> messaging versions.
>>> 
>>> You can continue upgrading, and when new nodes reach 3.0.14 they'll get the 
>>> new schema versions. Until they all match, schema changes won't propagate 
>>> properly.
>>> 
>>> If you REALLY need schema changes to propagate during the upgrade, you 
>>> should read the notes on CASSANDRA-13004 and Alex's summary here 
>>> https://gist.github.com/ifesdjeen/9cacb1ccd934374f707125d78f2fbcb6 
>>>  
>>> 
>>> But really, you should just upgrade to 3.0.15 and not do any schema changes 
>>> until the upgrade is complete (and be aware that having 2 schema versions 
>>> during the upgrade is normal and expected).
>>> 
>>> - Jeff
>>> 
>>> 
>>> On Wed, Nov 29, 2017 at 4:05 PM, Jai Bheemsen Rao Dhanwada 
>>> > wrote:
>>> Hello,
>>> 
>>> 
>>> 
>>> I am trying to upgrade the cassandra version from the 3.0.8 to 3.0.14. I am 
>>> adding a new node with 3.0.14 version to 3.0.8. cluster and I see the 
>>> schema disagreement between the nodes and the new node doesn't stream any 
>>> data.
>>> 
>>> I am looking at : https://issues.apache.org/jira/browse/CASSANDRA-13559 
>>> , does this mean, I 
>>> will not be able to add nodes with the higher version than 3.0.13?
>>> 
>>> here is what I see in the nodetool describecluster output
>>> 
>>> $ nodetool describecluster
>>> Cluster Information:
>>> Name: production
>>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> Schema versions:
>>> 45ad6427-30a8-3381-9e2c-266b446c6ea7: [192.168.1.2, 192.168.1.3, 
>>> 192.168.1.4]
>>> 
>>> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [10.10.1.10]
>>> Any work around to mitigate this?
>>> 
>>> 
>>> 
>> 
> 

Re: Schema version mismatch with 3.0.8 and 3.0.14

[Jai Bheemsen Rao Dhanwada]

Thanks Jeff,

I did some more testing on this version upgrade and here is brief summary
incase if someone runs into the same issue.

Basically, we cannot add new nodes with 3.0.14 or higher version to the
cluster with 3.0.8 version of cassandra (haven't tested with 3.0.13) and
below are the two options available

1. Rolling upgrade all the nodes in the cluster to 3.0.14 and add new nodes
with the same version 3.0.14
OR
2. use the flag "-Dcassandra.force_3_0_protocol_version=true" to join the
nodes with higher version.

On Thu, Nov 30, 2017 at 9:45 AM, Jeff Jirsa  wrote:

> I guess I didn’t realize you were adding and upgrading s
> at the same time
>
> Do one or the other first (either add the new node at the old version, or
> upgrade to the new version and then add the new node), or use the two step
> process described in 13004
>
> --
> Jeff Jirsa
>
>
> On Nov 30, 2017, at 9:40 AM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Hello Jeff,
>
> I tried with 3.0.15 as well and I see the same issue.
>
> Here are the steps I followed.
>
> 1. I have a 3 node C* cluster with 3.0.8 version
> 2. Added a new node (4th) node with 3.0.15 version of cassandra
> 3. New node joins the cluster but it is creates it's own schema and don't
> bootstrap any keyspace or column families from the cluster.
>
> in the logs I could see the below WARN message.
>
> OutboundTcpConnection.java:488 - Seed gossip version is 10; will not
> connect with that version
>
> so we cannot join a new node with the higher minor version to the cluster?
>
> On Wed, Nov 29, 2017 at 5:16 PM, Jeff Jirsa  wrote:
>
>> Yes, going to 3.0.14 or 3.0.15 should work correctly
>>
>> 3.0.15 will have important bug fixes not in 3.0.14
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Nov 29, 2017, at 4:52 PM, Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>> Thanks Jeff,
>>
>> You mean if I upgrade from 3.0.8 -> 3.0.15 it should work without any
>> issues?
>>
>> On Wed, Nov 29, 2017 at 4:21 PM, Jeff Jirsa  wrote:
>>
>>> This is because in 3.0.14, we had to bump the internode
>>> messaging version to fix a nasty bug (it's something we almost never do in
>>> minor versions, but it was necessary here), and schema changes won't cross
>>> internode messaging versions.
>>>
>>> You can continue upgrading, and when new nodes reach 3.0.14 they'll get
>>> the new schema versions. Until they all match, schema changes won't
>>> propagate properly.
>>>
>>> If you REALLY need schema changes to propagate during the upgrade, you
>>> should read the notes on CASSANDRA-13004 and Alex's summary here
>>> https://gist.github.com/ifesdjeen/9cacb1ccd934374f707125d78f2fbcb6
>>>
>>> But really, you should just upgrade to 3.0.15 and not do any schema
>>> changes until the upgrade is complete (and be aware that having 2 schema
>>> versions during the upgrade is normal and expected).
>>>
>>> - Jeff
>>>
>>>
>>> On Wed, Nov 29, 2017 at 4:05 PM, Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>>
 Hello,



 I am trying to upgrade the cassandra version from the 3.0.8 to 3.0.14.
 I am adding a new node with 3.0.14 version to 3.0.8. cluster and I see the
 schema disagreement between the nodes and the new node doesn't stream any
 data.

 I am looking at : https://issues.apache.org/jira/browse/CASSANDRA-13559,
 does this mean, I will not be able to add nodes with the higher version
 than 3.0.13?

 here is what I see in the nodetool describecluster output

 $ nodetool describecluster
 Cluster Information:
 Name: production
 Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
 Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
 Schema versions:
 45ad6427-30a8-3381-9e2c-266b446c6ea7: [192.168.1.2, 192.168.1.3, 
 192.168.1.4]

 c2a2bb4f-7d31-3fb8-a216-00b41a643650: [10.10.1.10]

 Any work around to mitigate this?

>>>
>>>
>>
>

Re: Schema version mismatch with 3.0.8 and 3.0.14

[Jeff Jirsa]

I guess I didn’t realize you were adding and upgrading s
at the same time

Do one or the other first (either add the new node at the old version, or 
upgrade to the new version and then add the new node), or use the two step 
process described in 13004

-- 
Jeff Jirsa


> On Nov 30, 2017, at 9:40 AM, Jai Bheemsen Rao Dhanwada 
>  wrote:
> 
> Hello Jeff,
> 
> I tried with 3.0.15 as well and I see the same issue.
> 
> Here are the steps I followed.
> 
> 1. I have a 3 node C* cluster with 3.0.8 version
> 2. Added a new node (4th) node with 3.0.15 version of cassandra
> 3. New node joins the cluster but it is creates it's own schema and don't 
> bootstrap any keyspace or column families from the cluster.
> 
> in the logs I could see the below WARN message.
> 
> OutboundTcpConnection.java:488 - Seed gossip version is 10; will not connect 
> with that version
> 
> so we cannot join a new node with the higher minor version to the cluster?
> 
>> On Wed, Nov 29, 2017 at 5:16 PM, Jeff Jirsa  wrote:
>> Yes, going to 3.0.14 or 3.0.15 should work correctly
>> 
>> 3.0.15 will have important bug fixes not in 3.0.14
>> 
>> 
>> -- 
>> Jeff Jirsa
>> 
>> 
>>> On Nov 29, 2017, at 4:52 PM, Jai Bheemsen Rao Dhanwada 
>>>  wrote:
>>> 
>>> Thanks Jeff,
>>> 
>>> You mean if I upgrade from 3.0.8 -> 3.0.15 it should work without any 
>>> issues?
>>> 
 On Wed, Nov 29, 2017 at 4:21 PM, Jeff Jirsa  wrote:
 This is because in 3.0.14, we had to bump the internode messaging version 
 to fix a nasty bug (it's something we almost never do in minor versions, 
 but it was necessary here), and schema changes won't cross internode 
 messaging versions.
 
 You can continue upgrading, and when new nodes reach 3.0.14 they'll get 
 the new schema versions. Until they all match, schema changes won't 
 propagate properly.
 
 If you REALLY need schema changes to propagate during the upgrade, you 
 should read the notes on CASSANDRA-13004 and Alex's summary here 
 https://gist.github.com/ifesdjeen/9cacb1ccd934374f707125d78f2fbcb6 
 
 But really, you should just upgrade to 3.0.15 and not do any schema 
 changes until the upgrade is complete (and be aware that having 2 schema 
 versions during the upgrade is normal and expected).
 
 - Jeff
 
 
> On Wed, Nov 29, 2017 at 4:05 PM, Jai Bheemsen Rao Dhanwada 
>  wrote:
> Hello,
> 
> 
> 
> I am trying to upgrade the cassandra version from the 3.0.8 to 3.0.14. I 
> am adding a new node with 3.0.14 version to 3.0.8. cluster and I see the 
> schema disagreement between the nodes and the new node doesn't stream any 
> data.
> 
> I am looking at : https://issues.apache.org/jira/browse/CASSANDRA-13559, 
> does this mean, I will not be able to add nodes with the higher version 
> than 3.0.13?
> 
> here is what I see in the nodetool describecluster output
> 
> $ nodetool describecluster
> Cluster Information:
> Name: production
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 45ad6427-30a8-3381-9e2c-266b446c6ea7: [192.168.1.2, 192.168.1.3, 
> 192.168.1.4]
> 
> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [10.10.1.10]
> Any work around to mitigate this?
> 
 
>>> 
> 

Re: Schema version mismatch with 3.0.8 and 3.0.14

[Jai Bheemsen Rao Dhanwada]

Hello Jeff,

I tried with 3.0.15 as well and I see the same issue.

Here are the steps I followed.

1. I have a 3 node C* cluster with 3.0.8 version
2. Added a new node (4th) node with 3.0.15 version of cassandra
3. New node joins the cluster but it is creates it's own schema and don't
bootstrap any keyspace or column families from the cluster.

in the logs I could see the below WARN message.

OutboundTcpConnection.java:488 - Seed gossip version is 10; will not
connect with that version

so we cannot join a new node with the higher minor version to the cluster?

On Wed, Nov 29, 2017 at 5:16 PM, Jeff Jirsa  wrote:

> Yes, going to 3.0.14 or 3.0.15 should work correctly
>
> 3.0.15 will have important bug fixes not in 3.0.14
>
>
> --
> Jeff Jirsa
>
>
> On Nov 29, 2017, at 4:52 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Thanks Jeff,
>
> You mean if I upgrade from 3.0.8 -> 3.0.15 it should work without any
> issues?
>
> On Wed, Nov 29, 2017 at 4:21 PM, Jeff Jirsa  wrote:
>
>> This is because in 3.0.14, we had to bump the internode messaging version
>> to fix a nasty bug (it's something we almost never do in minor versions,
>> but it was necessary here), and schema changes won't cross internode
>> messaging versions.
>>
>> You can continue upgrading, and when new nodes reach 3.0.14 they'll get
>> the new schema versions. Until they all match, schema changes won't
>> propagate properly.
>>
>> If you REALLY need schema changes to propagate during the upgrade, you
>> should read the notes on CASSANDRA-13004 and Alex's summary here
>> https://gist.github.com/ifesdjeen/9cacb1ccd934374f707125d78f2fbcb6
>>
>> But really, you should just upgrade to 3.0.15 and not do any schema
>> changes until the upgrade is complete (and be aware that having 2 schema
>> versions during the upgrade is normal and expected).
>>
>> - Jeff
>>
>>
>> On Wed, Nov 29, 2017 at 4:05 PM, Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>>
>>>
>>> I am trying to upgrade the cassandra version from the 3.0.8 to 3.0.14. I
>>> am adding a new node with 3.0.14 version to 3.0.8. cluster and I see the
>>> schema disagreement between the nodes and the new node doesn't stream any
>>> data.
>>>
>>> I am looking at : https://issues.apache.org/jira/browse/CASSANDRA-13559,
>>> does this mean, I will not be able to add nodes with the higher version
>>> than 3.0.13?
>>>
>>> here is what I see in the nodetool describecluster output
>>>
>>> $ nodetool describecluster
>>> Cluster Information:
>>> Name: production
>>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> Schema versions:
>>> 45ad6427-30a8-3381-9e2c-266b446c6ea7: [192.168.1.2, 192.168.1.3, 
>>> 192.168.1.4]
>>>
>>> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [10.10.1.10]
>>>
>>> Any work around to mitigate this?
>>>
>>
>>
>

Re: Schema version mismatch with 3.0.8 and 3.0.14

[Jeff Jirsa]

Yes, going to 3.0.14 or 3.0.15 should work correctly

3.0.15 will have important bug fixes not in 3.0.14


-- 
Jeff Jirsa


> On Nov 29, 2017, at 4:52 PM, Jai Bheemsen Rao Dhanwada 
>  wrote:
> 
> Thanks Jeff,
> 
> You mean if I upgrade from 3.0.8 -> 3.0.15 it should work without any issues?
> 
>> On Wed, Nov 29, 2017 at 4:21 PM, Jeff Jirsa  wrote:
>> This is because in 3.0.14, we had to bump the internode messaging version to 
>> fix a nasty bug (it's something we almost never do in minor versions, but it 
>> was necessary here), and schema changes won't cross internode messaging 
>> versions.
>> 
>> You can continue upgrading, and when new nodes reach 3.0.14 they'll get the 
>> new schema versions. Until they all match, schema changes won't propagate 
>> properly.
>> 
>> If you REALLY need schema changes to propagate during the upgrade, you 
>> should read the notes on CASSANDRA-13004 and Alex's summary here 
>> https://gist.github.com/ifesdjeen/9cacb1ccd934374f707125d78f2fbcb6 
>> 
>> But really, you should just upgrade to 3.0.15 and not do any schema changes 
>> until the upgrade is complete (and be aware that having 2 schema versions 
>> during the upgrade is normal and expected).
>> 
>> - Jeff
>> 
>> 
>>> On Wed, Nov 29, 2017 at 4:05 PM, Jai Bheemsen Rao Dhanwada 
>>>  wrote:
>>> Hello,
>>> 
>>> 
>>> 
>>> I am trying to upgrade the cassandra version from the 3.0.8 to 3.0.14. I am 
>>> adding a new node with 3.0.14 version to 3.0.8. cluster and I see the 
>>> schema disagreement between the nodes and the new node doesn't stream any 
>>> data.
>>> 
>>> I am looking at : https://issues.apache.org/jira/browse/CASSANDRA-13559, 
>>> does this mean, I will not be able to add nodes with the higher version 
>>> than 3.0.13?
>>> 
>>> here is what I see in the nodetool describecluster output
>>> 
>>> $ nodetool describecluster
>>> Cluster Information:
>>> Name: production
>>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> Schema versions:
>>> 45ad6427-30a8-3381-9e2c-266b446c6ea7: [192.168.1.2, 192.168.1.3, 
>>> 192.168.1.4]
>>> 
>>> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [10.10.1.10]
>>> Any work around to mitigate this?
>>> 
>> 
> 

Re: Schema version mismatch with 3.0.8 and 3.0.14

[Jai Bheemsen Rao Dhanwada]

Thanks Jeff,

You mean if I upgrade from 3.0.8 -> 3.0.15 it should work without any
issues?

On Wed, Nov 29, 2017 at 4:21 PM, Jeff Jirsa  wrote:

> This is because in 3.0.14, we had to bump the internode messaging version
> to fix a nasty bug (it's something we almost never do in minor versions,
> but it was necessary here), and schema changes won't cross internode
> messaging versions.
>
> You can continue upgrading, and when new nodes reach 3.0.14 they'll get
> the new schema versions. Until they all match, schema changes won't
> propagate properly.
>
> If you REALLY need schema changes to propagate during the upgrade, you
> should read the notes on CASSANDRA-13004 and Alex's summary here
> https://gist.github.com/ifesdjeen/9cacb1ccd934374f707125d78f2fbcb6
>
> But really, you should just upgrade to 3.0.15 and not do any schema
> changes until the upgrade is complete (and be aware that having 2 schema
> versions during the upgrade is normal and expected).
>
> - Jeff
>
>
> On Wed, Nov 29, 2017 at 4:05 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Hello,
>>
>>
>>
>> I am trying to upgrade the cassandra version from the 3.0.8 to 3.0.14. I
>> am adding a new node with 3.0.14 version to 3.0.8. cluster and I see the
>> schema disagreement between the nodes and the new node doesn't stream any
>> data.
>>
>> I am looking at : https://issues.apache.org/jira/browse/CASSANDRA-13559,
>> does this mean, I will not be able to add nodes with the higher version
>> than 3.0.13?
>>
>> here is what I see in the nodetool describecluster output
>>
>> $ nodetool describecluster
>> Cluster Information:
>> Name: production
>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Schema versions:
>> 45ad6427-30a8-3381-9e2c-266b446c6ea7: [192.168.1.2, 192.168.1.3, 
>> 192.168.1.4]
>>
>> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [10.10.1.10]
>>
>> Any work around to mitigate this?
>>
>
>

Re: Schema version mismatch with 3.0.8 and 3.0.14

[Jeff Jirsa]

This is because in 3.0.14, we had to bump the internode messaging version
to fix a nasty bug (it's something we almost never do in minor versions,
but it was necessary here), and schema changes won't cross internode
messaging versions.

You can continue upgrading, and when new nodes reach 3.0.14 they'll get the
new schema versions. Until they all match, schema changes won't propagate
properly.

If you REALLY need schema changes to propagate during the upgrade, you
should read the notes on CASSANDRA-13004 and Alex's summary here
https://gist.github.com/ifesdjeen/9cacb1ccd934374f707125d78f2fbcb6

But really, you should just upgrade to 3.0.15 and not do any schema changes
until the upgrade is complete (and be aware that having 2 schema versions
during the upgrade is normal and expected).

- Jeff


On Wed, Nov 29, 2017 at 4:05 PM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Hello,
>
>
>
> I am trying to upgrade the cassandra version from the 3.0.8 to 3.0.14. I
> am adding a new node with 3.0.14 version to 3.0.8. cluster and I see the
> schema disagreement between the nodes and the new node doesn't stream any
> data.
>
> I am looking at : https://issues.apache.org/jira/browse/CASSANDRA-13559,
> does this mean, I will not be able to add nodes with the higher version
> than 3.0.13?
>
> here is what I see in the nodetool describecluster output
>
> $ nodetool describecluster
> Cluster Information:
> Name: production
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 45ad6427-30a8-3381-9e2c-266b446c6ea7: [192.168.1.2, 192.168.1.3, 
> 192.168.1.4]
>
> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [10.10.1.10]
>
> Any work around to mitigate this?
>

Schema version mismatch with 3.0.8 and 3.0.14

[Jai Bheemsen Rao Dhanwada]

Hello,



I am trying to upgrade the cassandra version from the 3.0.8 to 3.0.14. I am
adding a new node with 3.0.14 version to 3.0.8. cluster and I see the
schema disagreement between the nodes and the new node doesn't stream any
data.

I am looking at : https://issues.apache.org/jira/browse/CASSANDRA-13559,
does this mean, I will not be able to add nodes with the higher version
than 3.0.13?

here is what I see in the nodetool describecluster output

$ nodetool describecluster
Cluster Information:
Name: production
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
45ad6427-30a8-3381-9e2c-266b446c6ea7: [192.168.1.2,
192.168.1.3, 192.168.1.4]

c2a2bb4f-7d31-3fb8-a216-00b41a643650: [10.10.1.10]

Any work around to mitigate this?

Re: Cassandra Schema version mismatch

[Nitan Kainth]

Yes Carlos same issue in our case. We had some cloud issue and node wasn’t 
marked down. We did have some schema changes a few days back but didn’t notice 
because one node was still serving and when that stopped serving, we saw issues.

So, it WAS due to schema changes that happened few days back and not propagated 
to all three replicas.

Thank you everybody for sharing their experience. 


> On May 6, 2017, at 7:15 AM, Carlos Rolo <r...@pythian.com> wrote:
> 
> Sometimes "resetlocalschema" (happened to me, didn't check why) will not 
> work, and you need to stop the offending nodes and bring them back one by 
> one.. That solved the issue.
> 
> In the cases I've seen this happened the clusters in question where either:
> 
> a) A couple of nodes down (cloud provider caused a partial outage) with an 
> application that would create/change tables every now and then
> b) Massive GC pauses around the cluster with a schema change being tried over 
> and over.
> 
> CQLSH stopped working, your nodes died? GC Pauses? Operator mistake?
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: 
> linkedin.com/in/carlosjuzarterolo 
>  <http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +351 918 918 100 
> www.pythian.com <http://www.pythian.com/>
> On Sat, May 6, 2017 at 1:07 PM, Nitan Kainth <ni...@bamlabs.com 
> <mailto:ni...@bamlabs.com>> wrote:
> Thank you Jeff.
> 
> James,
> 
> We started getting insufficient replica errors. Cqlsh stopped working for two 
> nodes .
> 
> Sent from my iPhone
> 
> > On May 5, 2017, at 7:13 PM, James Rothering <jrother...@codojo.me 
> > <mailto:jrother...@codojo.me>> wrote:
> >
> > I've heard about this ... how did the problem present itself?
> >
> > Sent from my iPhone
> >
> >> On May 5, 2017, at 3:17 PM, Jeff Jirsa <jji...@apache.org 
> >> <mailto:jji...@apache.org>> wrote:
> >>
> >>
> >>
> >>> On 2017-05-05 11:00 (-0700), Nitan Kainth <ni...@bamlabs.com 
> >>> <mailto:ni...@bamlabs.com>> wrote:
> >>> Hi Experts,
> >>>
> >>> We found schema version mismatch in our cluster. We fixed it by bouncing 
> >>> C* on nodes where version was mismatched. Can someone suggest, what are 
> >>> the possible reasons for this? We are trying to figure out the root cause.
> >>>
> >>
> >> Do all of your versions match? You didn't accidentally upgrade half the 
> >> cluster did you?
> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> >> <mailto:user-unsubscr...@cassandra.apache.org>
> >> For additional commands, e-mail: user-h...@cassandra.apache.org 
> >> <mailto:user-h...@cassandra.apache.org>
> >>
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> > <mailto:user-unsubscr...@cassandra.apache.org>
> > For additional commands, e-mail: user-h...@cassandra.apache.org 
> > <mailto:user-h...@cassandra.apache.org>
> >
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> <mailto:user-unsubscr...@cassandra.apache.org>
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> <mailto:user-h...@cassandra.apache.org>
> 
> 
> 
> --
> 
> 
> 
> 
> 

Re: Cassandra Schema version mismatch

[Carlos Rolo]

Sometimes "resetlocalschema" (happened to me, didn't check why) will not
work, and you need to stop the offending nodes and bring them back one by
one.. That solved the issue.

In the cases I've seen this happened the clusters in question where either:

a) A couple of nodes down (cloud provider caused a partial outage) with an
application that would create/change tables every now and then
b) Massive GC pauses around the cluster with a schema change being tried
over and over.

CQLSH stopped working, your nodes died? GC Pauses? Operator mistake?

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +351 918 918 100
www.pythian.com

On Sat, May 6, 2017 at 1:07 PM, Nitan Kainth <ni...@bamlabs.com> wrote:

> Thank you Jeff.
>
> James,
>
> We started getting insufficient replica errors. Cqlsh stopped working for
> two nodes .
>
> Sent from my iPhone
>
> > On May 5, 2017, at 7:13 PM, James Rothering <jrother...@codojo.me>
> wrote:
> >
> > I've heard about this ... how did the problem present itself?
> >
> > Sent from my iPhone
> >
> >> On May 5, 2017, at 3:17 PM, Jeff Jirsa <jji...@apache.org> wrote:
> >>
> >>
> >>
> >>> On 2017-05-05 11:00 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote:
> >>> Hi Experts,
> >>>
> >>> We found schema version mismatch in our cluster. We fixed it by
> bouncing C* on nodes where version was mismatched. Can someone suggest,
> what are the possible reasons for this? We are trying to figure out the
> root cause.
> >>>
> >>
> >> Do all of your versions match? You didn't accidentally upgrade half the
> cluster did you?
> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

-- 


--

Re: Cassandra Schema version mismatch

[Nitan Kainth]

Thank you Jeff.

James, 

We started getting insufficient replica errors. Cqlsh stopped working for two 
nodes .

Sent from my iPhone

> On May 5, 2017, at 7:13 PM, James Rothering <jrother...@codojo.me> wrote:
> 
> I've heard about this ... how did the problem present itself?
> 
> Sent from my iPhone
> 
>> On May 5, 2017, at 3:17 PM, Jeff Jirsa <jji...@apache.org> wrote:
>> 
>> 
>> 
>>> On 2017-05-05 11:00 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote: 
>>> Hi Experts,
>>> 
>>> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
>>> on nodes where version was mismatched. Can someone suggest, what are the 
>>> possible reasons for this? We are trying to figure out the root cause.
>>> 
>> 
>> Do all of your versions match? You didn't accidentally upgrade half the 
>> cluster did you?
>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Schema version mismatch

[James Rothering]

I've heard about this ... how did the problem present itself?

Sent from my iPhone

> On May 5, 2017, at 3:17 PM, Jeff Jirsa <jji...@apache.org> wrote:
> 
> 
> 
>> On 2017-05-05 11:00 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote: 
>> Hi Experts,
>> 
>> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
>> on nodes where version was mismatched. Can someone suggest, what are the 
>> possible reasons for this? We are trying to figure out the root cause.
>> 
> 
> Do all of your versions match? You didn't accidentally upgrade half the 
> cluster did you?
> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Schema version mismatch

[Jeff Jirsa]

Generally shouldn't happen in most modern versions of cassandra. Could be
simultaneous conflicting statements (two "CREATE TABLE" statements at the
same time, which can happen with programatic schema changes), or unhealthy
schema tables (lots and lots of changes create tombstones in the schema
tables, which can cause reading/calculating schema versions to be
inefficient).

In the future, you can work around this with 'nodetool resetlocalschema'



On Fri, May 5, 2017 at 3:45 PM, Nitan Kainth <ni...@bamlabs.com> wrote:

> No, just two nodes have mismatch out of 18 nodes. We upgraded long back.
>
> Sent from my iPhone
>
> > On May 5, 2017, at 5:17 PM, Jeff Jirsa <jji...@apache.org> wrote:
> >
> >
> >
> >> On 2017-05-05 11:00 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote:
> >> Hi Experts,
> >>
> >> We found schema version mismatch in our cluster. We fixed it by
> bouncing C* on nodes where version was mismatched. Can someone suggest,
> what are the possible reasons for this? We are trying to figure out the
> root cause.
> >>
> >
> > Do all of your versions match? You didn't accidentally upgrade half the
> cluster did you?
> >
> >
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>

Re: Cassandra Schema version mismatch

[Nitan Kainth]

No, just two nodes have mismatch out of 18 nodes. We upgraded long back. 

Sent from my iPhone

> On May 5, 2017, at 5:17 PM, Jeff Jirsa <jji...@apache.org> wrote:
> 
> 
> 
>> On 2017-05-05 11:00 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote: 
>> Hi Experts,
>> 
>> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
>> on nodes where version was mismatched. Can someone suggest, what are the 
>> possible reasons for this? We are trying to figure out the root cause.
>> 
> 
> Do all of your versions match? You didn't accidentally upgrade half the 
> cluster did you?
> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Schema version mismatch

[Jeff Jirsa]

On 2017-05-05 11:00 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote: 
> Hi Experts,
> 
> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
> on nodes where version was mismatched. Can someone suggest, what are the 
> possible reasons for this? We are trying to figure out the root cause.
> 

Do all of your versions match? You didn't accidentally upgrade half the cluster 
did you?



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Schema version mismatch

[Nitan Kainth]

No schematic change!

Sent from my iPhone

> On May 5, 2017, at 2:30 PM, Carlos Rolo <r...@pythian.com> wrote:
> 
> Are you changing the schema in a dynamic fashion? If you get problems 
> (network, gc pauses, etc) during the schema changes it might lead to that.
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: 
> linkedin.com/in/carlosjuzarterolo 
> Mobile: +351 918 918 100 
> www.pythian.com
> 
>> On Fri, May 5, 2017 at 7:00 PM, Nitan Kainth <ni...@bamlabs.com> wrote:
>> Hi Experts,
>> 
>> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
>> on nodes where version was mismatched. Can someone suggest, what are the 
>> possible reasons for this? We are trying to figure out the root cause.
>> 
>> thank you!
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> 
> --
> 
> 
> 
> 

Re: Cassandra Schema version mismatch

[Carlos Rolo]

Are you changing the schema in a dynamic fashion? If you get problems
(network, gc pauses, etc) during the schema changes it might lead to that.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +351 918 918 100
www.pythian.com

On Fri, May 5, 2017 at 7:00 PM, Nitan Kainth <ni...@bamlabs.com> wrote:

> Hi Experts,
>
> We found schema version mismatch in our cluster. We fixed it by bouncing
> C* on nodes where version was mismatched. Can someone suggest, what are the
> possible reasons for this? We are trying to figure out the root cause.
>
> thank you!
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

-- 


--

Cassandra Schema version mismatch

[Nitan Kainth]

Hi Experts,

We found schema version mismatch in our cluster. We fixed it by bouncing C* on 
nodes where version was mismatched. Can someone suggest, what are the possible 
reasons for this? We are trying to figure out the root cause.

thank you!
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org