Re: Cassandra Schema version mismatch

[Nitan Kainth]

Yes Carlos same issue in our case. We had some cloud issue and node wasn’t 
marked down. We did have some schema changes a few days back but didn’t notice 
because one node was still serving and when that stopped serving, we saw issues.

So, it WAS due to schema changes that happened few days back and not propagated 
to all three replicas.

Thank you everybody for sharing their experience. 


> On May 6, 2017, at 7:15 AM, Carlos Rolo  wrote:
> 
> Sometimes "resetlocalschema" (happened to me, didn't check why) will not 
> work, and you need to stop the offending nodes and bring them back one by 
> one.. That solved the issue.
> 
> In the cases I've seen this happened the clusters in question where either:
> 
> a) A couple of nodes down (cloud provider caused a partial outage) with an 
> application that would create/change tables every now and then
> b) Massive GC pauses around the cluster with a schema change being tried over 
> and over.
> 
> CQLSH stopped working, your nodes died? GC Pauses? Operator mistake?
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: 
> linkedin.com/in/carlosjuzarterolo 
>  
> Mobile: +351 918 918 100 
> www.pythian.com 
> On Sat, May 6, 2017 at 1:07 PM, Nitan Kainth  > wrote:
> Thank you Jeff.
> 
> James,
> 
> We started getting insufficient replica errors. Cqlsh stopped working for two 
> nodes .
> 
> Sent from my iPhone
> 
> > On May 5, 2017, at 7:13 PM, James Rothering  > > wrote:
> >
> > I've heard about this ... how did the problem present itself?
> >
> > Sent from my iPhone
> >
> >> On May 5, 2017, at 3:17 PM, Jeff Jirsa  >> > wrote:
> >>
> >>
> >>
> >>> On 2017-05-05 11:00 (-0700), Nitan Kainth  >>> > wrote:
> >>> Hi Experts,
> >>>
> >>> We found schema version mismatch in our cluster. We fixed it by bouncing 
> >>> C* on nodes where version was mismatched. Can someone suggest, what are 
> >>> the possible reasons for this? We are trying to figure out the root cause.
> >>>
> >>
> >> Do all of your versions match? You didn't accidentally upgrade half the 
> >> cluster did you?
> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> >> 
> >> For additional commands, e-mail: user-h...@cassandra.apache.org 
> >> 
> >>
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> > 
> > For additional commands, e-mail: user-h...@cassandra.apache.org 
> > 
> >
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 
> 
> 
> 
> --
> 
> 
> 
> 
> 

Re: Cassandra Schema version mismatch

[Carlos Rolo]

Sometimes "resetlocalschema" (happened to me, didn't check why) will not
work, and you need to stop the offending nodes and bring them back one by
one.. That solved the issue.

In the cases I've seen this happened the clusters in question where either:

a) A couple of nodes down (cloud provider caused a partial outage) with an
application that would create/change tables every now and then
b) Massive GC pauses around the cluster with a schema change being tried
over and over.

CQLSH stopped working, your nodes died? GC Pauses? Operator mistake?

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 918 918 100
www.pythian.com

On Sat, May 6, 2017 at 1:07 PM, Nitan Kainth  wrote:

> Thank you Jeff.
>
> James,
>
> We started getting insufficient replica errors. Cqlsh stopped working for
> two nodes .
>
> Sent from my iPhone
>
> > On May 5, 2017, at 7:13 PM, James Rothering 
> wrote:
> >
> > I've heard about this ... how did the problem present itself?
> >
> > Sent from my iPhone
> >
> >> On May 5, 2017, at 3:17 PM, Jeff Jirsa  wrote:
> >>
> >>
> >>
> >>> On 2017-05-05 11:00 (-0700), Nitan Kainth  wrote:
> >>> Hi Experts,
> >>>
> >>> We found schema version mismatch in our cluster. We fixed it by
> bouncing C* on nodes where version was mismatched. Can someone suggest,
> what are the possible reasons for this? We are trying to figure out the
> root cause.
> >>>
> >>
> >> Do all of your versions match? You didn't accidentally upgrade half the
> cluster did you?
> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

-- 


--

Re: Cassandra Schema version mismatch

[Nitan Kainth]

Thank you Jeff.

James, 

We started getting insufficient replica errors. Cqlsh stopped working for two 
nodes .

Sent from my iPhone

> On May 5, 2017, at 7:13 PM, James Rothering  wrote:
> 
> I've heard about this ... how did the problem present itself?
> 
> Sent from my iPhone
> 
>> On May 5, 2017, at 3:17 PM, Jeff Jirsa  wrote:
>> 
>> 
>> 
>>> On 2017-05-05 11:00 (-0700), Nitan Kainth  wrote: 
>>> Hi Experts,
>>> 
>>> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
>>> on nodes where version was mismatched. Can someone suggest, what are the 
>>> possible reasons for this? We are trying to figure out the root cause.
>>> 
>> 
>> Do all of your versions match? You didn't accidentally upgrade half the 
>> cluster did you?
>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Schema version mismatch

[James Rothering]

I've heard about this ... how did the problem present itself?

Sent from my iPhone

> On May 5, 2017, at 3:17 PM, Jeff Jirsa  wrote:
> 
> 
> 
>> On 2017-05-05 11:00 (-0700), Nitan Kainth  wrote: 
>> Hi Experts,
>> 
>> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
>> on nodes where version was mismatched. Can someone suggest, what are the 
>> possible reasons for this? We are trying to figure out the root cause.
>> 
> 
> Do all of your versions match? You didn't accidentally upgrade half the 
> cluster did you?
> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Schema version mismatch

[Jeff Jirsa]

Generally shouldn't happen in most modern versions of cassandra. Could be
simultaneous conflicting statements (two "CREATE TABLE" statements at the
same time, which can happen with programatic schema changes), or unhealthy
schema tables (lots and lots of changes create tombstones in the schema
tables, which can cause reading/calculating schema versions to be
inefficient).

In the future, you can work around this with 'nodetool resetlocalschema'



On Fri, May 5, 2017 at 3:45 PM, Nitan Kainth  wrote:

> No, just two nodes have mismatch out of 18 nodes. We upgraded long back.
>
> Sent from my iPhone
>
> > On May 5, 2017, at 5:17 PM, Jeff Jirsa  wrote:
> >
> >
> >
> >> On 2017-05-05 11:00 (-0700), Nitan Kainth  wrote:
> >> Hi Experts,
> >>
> >> We found schema version mismatch in our cluster. We fixed it by
> bouncing C* on nodes where version was mismatched. Can someone suggest,
> what are the possible reasons for this? We are trying to figure out the
> root cause.
> >>
> >
> > Do all of your versions match? You didn't accidentally upgrade half the
> cluster did you?
> >
> >
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>

Re: Cassandra Schema version mismatch

[Nitan Kainth]

No, just two nodes have mismatch out of 18 nodes. We upgraded long back. 

Sent from my iPhone

> On May 5, 2017, at 5:17 PM, Jeff Jirsa  wrote:
> 
> 
> 
>> On 2017-05-05 11:00 (-0700), Nitan Kainth  wrote: 
>> Hi Experts,
>> 
>> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
>> on nodes where version was mismatched. Can someone suggest, what are the 
>> possible reasons for this? We are trying to figure out the root cause.
>> 
> 
> Do all of your versions match? You didn't accidentally upgrade half the 
> cluster did you?
> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Schema version mismatch

[Jeff Jirsa]

On 2017-05-05 11:00 (-0700), Nitan Kainth  wrote: 
> Hi Experts,
> 
> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
> on nodes where version was mismatched. Can someone suggest, what are the 
> possible reasons for this? We are trying to figure out the root cause.
> 

Do all of your versions match? You didn't accidentally upgrade half the cluster 
did you?



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Schema version mismatch

[Nitan Kainth]

No schematic change!

Sent from my iPhone

> On May 5, 2017, at 2:30 PM, Carlos Rolo  wrote:
> 
> Are you changing the schema in a dynamic fashion? If you get problems 
> (network, gc pauses, etc) during the schema changes it might lead to that.
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: 
> linkedin.com/in/carlosjuzarterolo 
> Mobile: +351 918 918 100 
> www.pythian.com
> 
>> On Fri, May 5, 2017 at 7:00 PM, Nitan Kainth  wrote:
>> Hi Experts,
>> 
>> We found schema version mismatch in our cluster. We fixed it by bouncing C* 
>> on nodes where version was mismatched. Can someone suggest, what are the 
>> possible reasons for this? We are trying to figure out the root cause.
>> 
>> thank you!
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> 
> --
> 
> 
> 
> 

Re: Cassandra Schema version mismatch

[Carlos Rolo]

Are you changing the schema in a dynamic fashion? If you get problems
(network, gc pauses, etc) during the schema changes it might lead to that.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 918 918 100
www.pythian.com

On Fri, May 5, 2017 at 7:00 PM, Nitan Kainth  wrote:

> Hi Experts,
>
> We found schema version mismatch in our cluster. We fixed it by bouncing
> C* on nodes where version was mismatched. Can someone suggest, what are the
> possible reasons for this? We are trying to figure out the root cause.
>
> thank you!
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

-- 


--

Cassandra Schema version mismatch

[Nitan Kainth]

Hi Experts,

We found schema version mismatch in our cluster. We fixed it by bouncing C* on 
nodes where version was mismatched. Can someone suggest, what are the possible 
reasons for this? We are trying to figure out the root cause.

thank you!
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org