RE: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Jacques-Henri Berthemet]

You will have the same problem without IF NOT EXIST, at least I had Cassandra 
2.1 complaining about having tables with the same name but different UUIDs. In 
the end in our case we have a single application node that is responsible for 
schema upgrades, that’s ok for us as we don’t plan to have the schema upgraded 
that much.

--
Jacques-Henri Berthemet

From: Ken Hancock [mailto:ken.hanc...@schange.com]
Sent: mardi 2 février 2016 17:14
To: user@cassandra.apache.org
Subject: Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 
upgrade

Just to close the loop on this, but am I correct that the IF NOT EXITS isn't 
the real problem?  Even multiple calls to CREATE TABLE cause the same schema 
mismatch if done concurrently?  Normally, a CREATE TABLE call will return an 
exception that the table already exists.

On Tue, Feb 2, 2016 at 11:06 AM, Jack Krupansky 
<jack.krupan...@gmail.com<mailto:jack.krupan...@gmail.com>> wrote:
And CASSANDRA-10699  seems to be the sub-issue of CASSANDRA-9424 to do that:
https://issues.apache.org/jira/browse/CASSANDRA-10699


-- Jack Krupansky

On Tue, Feb 2, 2016 at 9:59 AM, Sebastian Estevez 
<sebastian.este...@datastax.com<mailto:sebastian.este...@datastax.com>> wrote:

Hi Ken,

Earlier in this thread I posted a link to 
https://issues.apache.org/jira/browse/CASSANDRA-9424

That is the fix for these schema disagreement issues and as ay commented, the 
plan is to use CAS. Until then we have to treat schema delicately.

all the best,

Sebastián
On Feb 2, 2016 9:48 AM, "Ken Hancock" 
<ken.hanc...@schange.com<mailto:ken.hanc...@schange.com>> wrote:
So this rings odd to me.  If you can accomplish the same thing by using a CAS 
operation, why not fix create table if not exist so that if your are writing an 
application that creates the table on startup, that the application is safe to 
run on multiple nodes and uses CAS to safeguard multiple concurrent creations?

On Tue, Jan 26, 2016 at 12:32 PM, Eric Stevens 
<migh...@gmail.com<mailto:migh...@gmail.com>> wrote:
There's still a race condition there, because two clients could SELECT at the 
same time as each other, then both INSERT.

You'd be better served with a CAS operation, and let Paxos guarantee 
at-most-once execution.

On Tue, Jan 26, 2016 at 9:06 AM Francisco Reyes 
<li...@natserv.net<mailto:li...@natserv.net>> wrote:
On 01/22/2016 10:29 PM, Kevin Burton wrote:
I sort of agree.. but we are also considering migrating to hourly tables.. and 
what if the single script doesn't run.

I like having N nodes make changes like this because in my experience that 
central / single box will usually fail at the wrong time :-/



On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
<j...@jonhaddad.com<mailto:j...@jonhaddad.com>> wrote:
Instead of using ZK, why not solve your concurrency problem by removing it?  By 
that, I mean simply have 1 process that creates all your tables instead of 
creating a race condition intentionally?

On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton 
<bur...@spinn3r.com<mailto:bur...@spinn3r.com>> wrote:
Not sure if this is a bug or not or kind of a *fuzzy* area.

In 2.0 this worked fine.

We have a bunch of automated scripts that go through and create tables... one 
per day.

at midnight UTC our entire CQL went offline.. .took down our whole app.  ;-/

The resolution was a full CQL shut down and then a drop table to remove the bad 
tables...

pretty sure the issue was with schema disagreement.

All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT EXISTS only 
checks locally?

My work around is going to be to use zookeeper to create a mutex lock during 
this operation.

Any other things I should avoid?


--
We’re hiring if you know of any awesome Java Devops or Linux Operations 
Engineers!

Founder/CEO Spinn3r.com<http://Spinn3r.com>
Location: San Francisco, CA
blog: http://burtonator.wordpress.com
… or check out my Google+ 
profile<https://plus.google.com/102718274791889610666/posts>
Error! Filename not specified.



--
We’re hiring if you know of any awesome Java Devops or Linux Operations 
Engineers!

Founder/CEO Spinn3r.com<http://Spinn3r.com>
Location: San Francisco, CA
blog: http://burtonator.wordpress.com
… or check out my Google+ 
profile<https://plus.google.com/102718274791889610666/posts>
Error! Filename not specified.

One way to accomplish both, a single process doing the work and having multiple 
machines be able to do it, is to have a control table.

You can have a table that lists what tables have been created and force 
concistency all. In this table you list the names of tables created. If a table 
name is in there, it doesn't need to be created again.



--
Ken Hancock | System Architect, Advanced Advertising
SeaChange International
50 Nagog Park
Acton, Massachusetts 01720
ken.hanc...@schange.com<mailto:ken.hanc...@schange.com> | 
www.s

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Sebastian Estevez]

Hi Ken,

Earlier in this thread I posted a link to
https://issues.apache.org/jira/browse/CASSANDRA-9424

That is the fix for these schema disagreement issues and as ay commented,
the plan is to use CAS. Until then we have to treat schema delicately.

all the best,

Sebastián
On Feb 2, 2016 9:48 AM, "Ken Hancock"  wrote:

> So this rings odd to me.  If you can accomplish the same thing by using a
> CAS operation, why not fix create table if not exist so that if your are
> writing an application that creates the table on startup, that the
> application is safe to run on multiple nodes and uses CAS to safeguard
> multiple concurrent creations?
>
>
> On Tue, Jan 26, 2016 at 12:32 PM, Eric Stevens  wrote:
>
>> There's still a race condition there, because two clients could SELECT at
>> the same time as each other, then both INSERT.
>>
>> You'd be better served with a CAS operation, and let Paxos guarantee
>> at-most-once execution.
>>
>> On Tue, Jan 26, 2016 at 9:06 AM Francisco Reyes 
>> wrote:
>>
>>> On 01/22/2016 10:29 PM, Kevin Burton wrote:
>>>
>>> I sort of agree.. but we are also considering migrating to hourly
>>> tables.. and what if the single script doesn't run.
>>>
>>> I like having N nodes make changes like this because in my experience
>>> that central / single box will usually fail at the wrong time :-/
>>>
>>>
>>>
>>> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
>>> wrote:
>>>
 Instead of using ZK, why not solve your concurrency problem by removing
 it?  By that, I mean simply have 1 process that creates all your tables
 instead of creating a race condition intentionally?

 On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton 
 wrote:

> Not sure if this is a bug or not or kind of a *fuzzy* area.
>
> In 2.0 this worked fine.
>
> We have a bunch of automated scripts that go through and create
> tables... one per day.
>
> at midnight UTC our entire CQL went offline.. .took down our whole
> app.  ;-/
>
> The resolution was a full CQL shut down and then a drop table to
> remove the bad tables...
>
> pretty sure the issue was with schema disagreement.
>
> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT
> EXISTS only checks locally?
>
> My work around is going to be to use zookeeper to create a mutex lock
> during this operation.
>
> Any other things I should avoid?
>
>
> --
> We’re hiring if you know of any awesome Java Devops or Linux
> Operations Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog:  
> http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>
>>>
>>>
>>> --
>>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>>> Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog:  http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> 
>>>
>>>
>>> One way to accomplish both, a single process doing the work and having
>>> multiple machines be able to do it, is to have a control table.
>>>
>>> You can have a table that lists what tables have been created and force
>>> concistency all. In this table you list the names of tables created. If a
>>> table name is in there, it doesn't need to be created again.
>>>
>>
>
>
> --
> *Ken Hancock *| System Architect, Advanced Advertising
> SeaChange International
> 50 Nagog Park
> Acton, Massachusetts 01720
> ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC
> 
> Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com
>  | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
> LinkedIn] 
>
> [image: SeaChange International]
> 
> This e-mail and any attachments may contain information which is SeaChange
> International confidential. The information enclosed is intended only for
> the addressees herein and may not be copied or forwarded without permission
> from SeaChange International.
>

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Ken Hancock]

So this rings odd to me.  If you can accomplish the same thing by using a
CAS operation, why not fix create table if not exist so that if your are
writing an application that creates the table on startup, that the
application is safe to run on multiple nodes and uses CAS to safeguard
multiple concurrent creations?


On Tue, Jan 26, 2016 at 12:32 PM, Eric Stevens  wrote:

> There's still a race condition there, because two clients could SELECT at
> the same time as each other, then both INSERT.
>
> You'd be better served with a CAS operation, and let Paxos guarantee
> at-most-once execution.
>
> On Tue, Jan 26, 2016 at 9:06 AM Francisco Reyes  wrote:
>
>> On 01/22/2016 10:29 PM, Kevin Burton wrote:
>>
>> I sort of agree.. but we are also considering migrating to hourly
>> tables.. and what if the single script doesn't run.
>>
>> I like having N nodes make changes like this because in my experience
>> that central / single box will usually fail at the wrong time :-/
>>
>>
>>
>> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
>> wrote:
>>
>>> Instead of using ZK, why not solve your concurrency problem by removing
>>> it?  By that, I mean simply have 1 process that creates all your tables
>>> instead of creating a race condition intentionally?
>>>
>>> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton  wrote:
>>>
 Not sure if this is a bug or not or kind of a *fuzzy* area.

 In 2.0 this worked fine.

 We have a bunch of automated scripts that go through and create
 tables... one per day.

 at midnight UTC our entire CQL went offline.. .took down our whole app.
  ;-/

 The resolution was a full CQL shut down and then a drop table to remove
 the bad tables...

 pretty sure the issue was with schema disagreement.

 All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT
 EXISTS only checks locally?

 My work around is going to be to use zookeeper to create a mutex lock
 during this operation.

 Any other things I should avoid?


 --
 We’re hiring if you know of any awesome Java Devops or Linux Operations
 Engineers!

 Founder/CEO Spinn3r.com
 Location: *San Francisco, CA*
 blog:  http://burtonator.wordpress.com
 … or check out my Google+ profile
 


>>
>>
>> --
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog:  http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>>
>>
>> One way to accomplish both, a single process doing the work and having
>> multiple machines be able to do it, is to have a control table.
>>
>> You can have a table that lists what tables have been created and force
>> concistency all. In this table you list the names of tables created. If a
>> table name is in there, it doesn't need to be created again.
>>
>


-- 
*Ken Hancock *| System Architect, Advanced Advertising
SeaChange International
50 Nagog Park
Acton, Massachusetts 01720
ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC

Office: +1 (978) 889-3329 | [image: Google Talk:]
ken.hanc...@schange.com | [image:
Skype:]hancockks | [image: Yahoo IM:]hancockks [image: LinkedIn]


[image: SeaChange International]

This e-mail and any attachments may contain information which is SeaChange
International confidential. The information enclosed is intended only for
the addressees herein and may not be copied or forwarded without permission
from SeaChange International.

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Ken Hancock]

Just to close the loop on this, but am I correct that the IF NOT EXITS
isn't the real problem?  Even multiple calls to CREATE TABLE cause the same
schema mismatch if done concurrently?  Normally, a CREATE TABLE call will
return an exception that the table already exists.

On Tue, Feb 2, 2016 at 11:06 AM, Jack Krupansky 
wrote:

> And CASSANDRA-10699  seems to be the sub-issue of CASSANDRA-9424 to do
> that:
> https://issues.apache.org/jira/browse/CASSANDRA-10699
>
>
> -- Jack Krupansky
>
> On Tue, Feb 2, 2016 at 9:59 AM, Sebastian Estevez <
> sebastian.este...@datastax.com> wrote:
>
>> Hi Ken,
>>
>> Earlier in this thread I posted a link to
>> https://issues.apache.org/jira/browse/CASSANDRA-9424
>>
>> That is the fix for these schema disagreement issues and as ay commented,
>> the plan is to use CAS. Until then we have to treat schema delicately.
>>
>> all the best,
>>
>> Sebastián
>> On Feb 2, 2016 9:48 AM, "Ken Hancock"  wrote:
>>
>>> So this rings odd to me.  If you can accomplish the same thing by using
>>> a CAS operation, why not fix create table if not exist so that if your are
>>> writing an application that creates the table on startup, that the
>>> application is safe to run on multiple nodes and uses CAS to safeguard
>>> multiple concurrent creations?
>>>
>>>
>>> On Tue, Jan 26, 2016 at 12:32 PM, Eric Stevens 
>>> wrote:
>>>
 There's still a race condition there, because two clients could SELECT
 at the same time as each other, then both INSERT.

 You'd be better served with a CAS operation, and let Paxos guarantee
 at-most-once execution.

 On Tue, Jan 26, 2016 at 9:06 AM Francisco Reyes 
 wrote:

> On 01/22/2016 10:29 PM, Kevin Burton wrote:
>
> I sort of agree.. but we are also considering migrating to hourly
> tables.. and what if the single script doesn't run.
>
> I like having N nodes make changes like this because in my experience
> that central / single box will usually fail at the wrong time :-/
>
>
>
> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
> wrote:
>
>> Instead of using ZK, why not solve your concurrency problem by
>> removing it?  By that, I mean simply have 1 process that creates all your
>> tables instead of creating a race condition intentionally?
>>
>> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton 
>> wrote:
>>
>>> Not sure if this is a bug or not or kind of a *fuzzy* area.
>>>
>>> In 2.0 this worked fine.
>>>
>>> We have a bunch of automated scripts that go through and create
>>> tables... one per day.
>>>
>>> at midnight UTC our entire CQL went offline.. .took down our whole
>>> app.  ;-/
>>>
>>> The resolution was a full CQL shut down and then a drop table to
>>> remove the bad tables...
>>>
>>> pretty sure the issue was with schema disagreement.
>>>
>>> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT
>>> EXISTS only checks locally?
>>>
>>> My work around is going to be to use zookeeper to create a mutex
>>> lock during this operation.
>>>
>>> Any other things I should avoid?
>>>
>>>
>>> --
>>> We’re hiring if you know of any awesome Java Devops or Linux
>>> Operations Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog:  
>>> http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> 
>>>
>>>
>
>
> --
> We’re hiring if you know of any awesome Java Devops or Linux
> Operations Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog:  
> http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>
> One way to accomplish both, a single process doing the work and having
> multiple machines be able to do it, is to have a control table.
>
> You can have a table that lists what tables have been created and
> force concistency all. In this table you list the names of tables created.
> If a table name is in there, it doesn't need to be created again.
>

>>>
>>>
>>> --
>>> *Ken Hancock *| System Architect, Advanced Advertising
>>> SeaChange International
>>> 50 Nagog Park
>>> Acton, Massachusetts 01720
>>> ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC
>>> 
>>> Office: +1 (978) 889-3329 | [image: Google Talk:]
>>> ken.hanc...@schange.com | [image: Skype:]hancockks | [image: Yahoo IM:]
>>> hancockks [image: LinkedIn] 
>>>
>>> 

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Jack Krupansky]

And CASSANDRA-10699  seems to be the sub-issue of CASSANDRA-9424 to do that:
https://issues.apache.org/jira/browse/CASSANDRA-10699


-- Jack Krupansky

On Tue, Feb 2, 2016 at 9:59 AM, Sebastian Estevez <
sebastian.este...@datastax.com> wrote:

> Hi Ken,
>
> Earlier in this thread I posted a link to
> https://issues.apache.org/jira/browse/CASSANDRA-9424
>
> That is the fix for these schema disagreement issues and as ay commented,
> the plan is to use CAS. Until then we have to treat schema delicately.
>
> all the best,
>
> Sebastián
> On Feb 2, 2016 9:48 AM, "Ken Hancock"  wrote:
>
>> So this rings odd to me.  If you can accomplish the same thing by using a
>> CAS operation, why not fix create table if not exist so that if your are
>> writing an application that creates the table on startup, that the
>> application is safe to run on multiple nodes and uses CAS to safeguard
>> multiple concurrent creations?
>>
>>
>> On Tue, Jan 26, 2016 at 12:32 PM, Eric Stevens  wrote:
>>
>>> There's still a race condition there, because two clients could SELECT
>>> at the same time as each other, then both INSERT.
>>>
>>> You'd be better served with a CAS operation, and let Paxos guarantee
>>> at-most-once execution.
>>>
>>> On Tue, Jan 26, 2016 at 9:06 AM Francisco Reyes 
>>> wrote:
>>>
 On 01/22/2016 10:29 PM, Kevin Burton wrote:

 I sort of agree.. but we are also considering migrating to hourly
 tables.. and what if the single script doesn't run.

 I like having N nodes make changes like this because in my experience
 that central / single box will usually fail at the wrong time :-/



 On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
 wrote:

> Instead of using ZK, why not solve your concurrency problem by
> removing it?  By that, I mean simply have 1 process that creates all your
> tables instead of creating a race condition intentionally?
>
> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton 
> wrote:
>
>> Not sure if this is a bug or not or kind of a *fuzzy* area.
>>
>> In 2.0 this worked fine.
>>
>> We have a bunch of automated scripts that go through and create
>> tables... one per day.
>>
>> at midnight UTC our entire CQL went offline.. .took down our whole
>> app.  ;-/
>>
>> The resolution was a full CQL shut down and then a drop table to
>> remove the bad tables...
>>
>> pretty sure the issue was with schema disagreement.
>>
>> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT
>> EXISTS only checks locally?
>>
>> My work around is going to be to use zookeeper to create a mutex lock
>> during this operation.
>>
>> Any other things I should avoid?
>>
>>
>> --
>> We’re hiring if you know of any awesome Java Devops or Linux
>> Operations Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog:  
>> http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>>
>>


 --
 We’re hiring if you know of any awesome Java Devops or Linux Operations
 Engineers!

 Founder/CEO Spinn3r.com
 Location: *San Francisco, CA*
 blog:  http://burtonator.wordpress.com
 … or check out my Google+ profile
 


 One way to accomplish both, a single process doing the work and having
 multiple machines be able to do it, is to have a control table.

 You can have a table that lists what tables have been created and force
 concistency all. In this table you list the names of tables created. If a
 table name is in there, it doesn't need to be created again.

>>>
>>
>>
>> --
>> *Ken Hancock *| System Architect, Advanced Advertising
>> SeaChange International
>> 50 Nagog Park
>> Acton, Massachusetts 01720
>> ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC
>> 
>> Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com
>>  | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
>> LinkedIn] 
>>
>> [image: SeaChange International]
>> 
>> This e-mail and any attachments may contain information which is
>> SeaChange International confidential. The information enclosed is intended
>> only for the addressees herein and may not be copied or forwarded without
>> permission from SeaChange International.
>>
>

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Francisco Reyes]

On 01/22/2016 10:29 PM, Kevin Burton wrote:
I sort of agree.. but we are also considering migrating to hourly 
tables.. and what if the single script doesn't run.


I like having N nodes make changes like this because in my experience 
that central / single box will usually fail at the wrong time :-/




On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad > wrote:


Instead of using ZK, why not solve your concurrency problem by
removing it?  By that, I mean simply have 1 process that creates
all your tables instead of creating a race condition intentionally?

On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton > wrote:

Not sure if this is a bug or not or kind of a *fuzzy* area.

In 2.0 this worked fine.

We have a bunch of automated scripts that go through and
create tables... one per day.

at midnight UTC our entire CQL went offline.. .took down our
whole app.  ;-/

The resolution was a full CQL shut down and then a drop table
to remove the bad tables...

pretty sure the issue was with schema disagreement.

All our CREATE TABLE use IF NOT EXISTS but I think the IF
NOT EXISTS only checks locally?

My work around is going to be to use zookeeper to create a
mutex lock during this operation.

Any other things I should avoid?


-- 
We’re hiring if you know of any awesome Java Devops or Linux

Operations Engineers!

Founder/CEO Spinn3r.com 
Location: *San Francisco, CA*
blog:**http://burtonator.wordpress.com
… or check out my Google+ profile





--
We’re hiring if you know of any awesome Java Devops or Linux 
Operations Engineers!


Founder/CEO Spinn3r.com 
Location: *San Francisco, CA*
blog:**http://burtonator.wordpress.com
… or check out my Google+ profile 





One way to accomplish both, a single process doing the work and having 
multiple machines be able to do it, is to have a control table.


You can have a table that lists what tables have been created and force 
concistency all. In this table you list the names of tables created. If 
a table name is in there, it doesn't need to be created again.

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Eric Stevens]

There's still a race condition there, because two clients could SELECT at
the same time as each other, then both INSERT.

You'd be better served with a CAS operation, and let Paxos guarantee
at-most-once execution.

On Tue, Jan 26, 2016 at 9:06 AM Francisco Reyes  wrote:

> On 01/22/2016 10:29 PM, Kevin Burton wrote:
>
> I sort of agree.. but we are also considering migrating to hourly tables..
> and what if the single script doesn't run.
>
> I like having N nodes make changes like this because in my experience that
> central / single box will usually fail at the wrong time :-/
>
>
>
> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
> wrote:
>
>> Instead of using ZK, why not solve your concurrency problem by removing
>> it?  By that, I mean simply have 1 process that creates all your tables
>> instead of creating a race condition intentionally?
>>
>> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton  wrote:
>>
>>> Not sure if this is a bug or not or kind of a *fuzzy* area.
>>>
>>> In 2.0 this worked fine.
>>>
>>> We have a bunch of automated scripts that go through and create
>>> tables... one per day.
>>>
>>> at midnight UTC our entire CQL went offline.. .took down our whole app.
>>>  ;-/
>>>
>>> The resolution was a full CQL shut down and then a drop table to remove
>>> the bad tables...
>>>
>>> pretty sure the issue was with schema disagreement.
>>>
>>> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT EXISTS
>>> only checks locally?
>>>
>>> My work around is going to be to use zookeeper to create a mutex lock
>>> during this operation.
>>>
>>> Any other things I should avoid?
>>>
>>>
>>> --
>>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>>> Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> 
>>>
>>>
>
>
> --
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>
> One way to accomplish both, a single process doing the work and having
> multiple machines be able to do it, is to have a control table.
>
> You can have a table that lists what tables have been created and force
> concistency all. In this table you list the names of tables created. If a
> table name is in there, it doesn't need to be created again.
>

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Sebastian Estevez]

You have to wait for schema agreement which most drivers should do by
default. At least have a check schema agreement method you can use.

https://datastax.github.io/java-driver/2.1.9/features/metadata/

The new cqlsh uses the python driver so the same should apply:

https://datastax.github.io/python-driver/api/cassandra/cluster.html

But check 'nodetool describecluster' to confirm that all nodes have the
same schema version.

Note: This will not help you in the concurrency / multiple writers
scenario.

all the best,

Sebastián
On Jan 23, 2016 7:29 PM, "Kevin Burton"  wrote:

> Once the CREATE TABLE returns in cqlsh (or programatically) is it safe to
> assume it's on all nodes at that point?
>
> If not I'll have to put in even more logic to handle this case..
>
> On Fri, Jan 22, 2016 at 9:22 PM, Jack Krupansky 
> wrote:
>
>> I recall that there was some discussion last year about this issue of how
>> risky it is to do an automated CREATE TABLE IF NOT EXISTS due to the
>> unpredictable amount of time it takes for the table creation to fully
>> propagate around the full cluster. I think it was recognized as a real
>> problem, but without an immediate solution, so the recommended practice for
>> now is to only manually perform the operation (sure, it can be scripted,
>> but only under manual control) to assure that the operation completes and
>> that only one attempt is made to create the table. I don't recall if there
>> was a specific Jira assigned, and the antipattern doc doesn't appear to
>> reference this scenario. Maybe a committer can shed some more light.
>>
>> -- Jack Krupansky
>>
>> On Fri, Jan 22, 2016 at 10:29 PM, Kevin Burton 
>> wrote:
>>
>>> I sort of agree.. but we are also considering migrating to hourly
>>> tables.. and what if the single script doesn't run.
>>>
>>> I like having N nodes make changes like this because in my experience
>>> that central / single box will usually fail at the wrong time :-/
>>>
>>>
>>>
>>> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
>>> wrote:
>>>
 Instead of using ZK, why not solve your concurrency problem by removing
 it?  By that, I mean simply have 1 process that creates all your tables
 instead of creating a race condition intentionally?

 On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton 
 wrote:

> Not sure if this is a bug or not or kind of a *fuzzy* area.
>
> In 2.0 this worked fine.
>
> We have a bunch of automated scripts that go through and create
> tables... one per day.
>
> at midnight UTC our entire CQL went offline.. .took down our whole
> app.  ;-/
>
> The resolution was a full CQL shut down and then a drop table to
> remove the bad tables...
>
> pretty sure the issue was with schema disagreement.
>
> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT
> EXISTS only checks locally?
>
> My work around is going to be to use zookeeper to create a mutex lock
> during this operation.
>
> Any other things I should avoid?
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux
> Operations Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>
>>>
>>>
>>> --
>>>
>>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>>> Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> 
>>>
>>>
>>
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Eric Stevens]

It seems like this exact problem pops up every few weeks on this list.  I
think the documentation does a dangerously bad job of describing the
limitations of CREATE TABLE...IF NOT EXISTS.

CREATE TABLE...IF NOT EXISTS is a dangerous construct because it seems to
advertise atomicity and isolation, neither of which it actually possesses.
Worse, the failure mode isn't just unpredictable output, but outright
failure of cluster stability.  The documentation on this does not do an
adequate job of describing what it actually does, and its characteristics
are inconsistent with other forms of IF NOT EXISTS.

> Cassandra 2.1.1 and later supports the IF NOT EXISTS syntax for creating
a trigger [SIC]. Attempting to create an existing table returns an error
unless the IF NOT EXISTS option is used. If the option is used, the
statement if a no-op if the table already exists.

I would strongly suggest this documentation be updated to indicate that it
is NOT SAFE to rely on atomicity and isolation of this statement, and that
it cannot be used like relational databases to coordinate schema changes.


On Sat, Jan 23, 2016 at 5:29 PM Kevin Burton  wrote:

> Once the CREATE TABLE returns in cqlsh (or programatically) is it safe to
> assume it's on all nodes at that point?
>
> If not I'll have to put in even more logic to handle this case..
>
> On Fri, Jan 22, 2016 at 9:22 PM, Jack Krupansky 
> wrote:
>
>> I recall that there was some discussion last year about this issue of how
>> risky it is to do an automated CREATE TABLE IF NOT EXISTS due to the
>> unpredictable amount of time it takes for the table creation to fully
>> propagate around the full cluster. I think it was recognized as a real
>> problem, but without an immediate solution, so the recommended practice for
>> now is to only manually perform the operation (sure, it can be scripted,
>> but only under manual control) to assure that the operation completes and
>> that only one attempt is made to create the table. I don't recall if there
>> was a specific Jira assigned, and the antipattern doc doesn't appear to
>> reference this scenario. Maybe a committer can shed some more light.
>>
>> -- Jack Krupansky
>>
>> On Fri, Jan 22, 2016 at 10:29 PM, Kevin Burton 
>> wrote:
>>
>>> I sort of agree.. but we are also considering migrating to hourly
>>> tables.. and what if the single script doesn't run.
>>>
>>> I like having N nodes make changes like this because in my experience
>>> that central / single box will usually fail at the wrong time :-/
>>>
>>>
>>>
>>> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
>>> wrote:
>>>
 Instead of using ZK, why not solve your concurrency problem by removing
 it?  By that, I mean simply have 1 process that creates all your tables
 instead of creating a race condition intentionally?

 On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton 
 wrote:

> Not sure if this is a bug or not or kind of a *fuzzy* area.
>
> In 2.0 this worked fine.
>
> We have a bunch of automated scripts that go through and create
> tables... one per day.
>
> at midnight UTC our entire CQL went offline.. .took down our whole
> app.  ;-/
>
> The resolution was a full CQL shut down and then a drop table to
> remove the bad tables...
>
> pretty sure the issue was with schema disagreement.
>
> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT
> EXISTS only checks locally?
>
> My work around is going to be to use zookeeper to create a mutex lock
> during this operation.
>
> Any other things I should avoid?
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux
> Operations Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>
>>>
>>>
>>> --
>>>
>>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>>> Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> 
>>>
>>>
>>
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Jack Krupansky]

+1 for doc update. I added some comments to the seemingly most relevant
Jira ticket to confirm best practice that we can then forward to the doc
team:
https://issues.apache.org/jira/browse/CASSANDRA-10699


-- Jack Krupansky

On Mon, Jan 25, 2016 at 1:12 PM, Eric Stevens  wrote:

> It seems like this exact problem pops up every few weeks on this list.  I
> think the documentation does a dangerously bad job of describing the
> limitations of CREATE TABLE...IF NOT EXISTS.
>
> CREATE TABLE...IF NOT EXISTS is a dangerous construct because it seems to
> advertise atomicity and isolation, neither of which it actually possesses.
> Worse, the failure mode isn't just unpredictable output, but outright
> failure of cluster stability.  The documentation on this does not do an
> adequate job of describing what it actually does, and its characteristics
> are inconsistent with other forms of IF NOT EXISTS.
>
> > Cassandra 2.1.1 and later supports the IF NOT EXISTS syntax for
> creating a trigger [SIC]. Attempting to create an existing table returns an
> error unless the IF NOT EXISTS option is used. If the option is used, the
> statement if a no-op if the table already exists.
>
> I would strongly suggest this documentation be updated to indicate that it
> is NOT SAFE to rely on atomicity and isolation of this statement, and that
> it cannot be used like relational databases to coordinate schema changes.
>
>
> On Sat, Jan 23, 2016 at 5:29 PM Kevin Burton  wrote:
>
>> Once the CREATE TABLE returns in cqlsh (or programatically) is it safe to
>> assume it's on all nodes at that point?
>>
>> If not I'll have to put in even more logic to handle this case..
>>
>> On Fri, Jan 22, 2016 at 9:22 PM, Jack Krupansky > > wrote:
>>
>>> I recall that there was some discussion last year about this issue of
>>> how risky it is to do an automated CREATE TABLE IF NOT EXISTS due to the
>>> unpredictable amount of time it takes for the table creation to fully
>>> propagate around the full cluster. I think it was recognized as a real
>>> problem, but without an immediate solution, so the recommended practice for
>>> now is to only manually perform the operation (sure, it can be scripted,
>>> but only under manual control) to assure that the operation completes and
>>> that only one attempt is made to create the table. I don't recall if there
>>> was a specific Jira assigned, and the antipattern doc doesn't appear to
>>> reference this scenario. Maybe a committer can shed some more light.
>>>
>>> -- Jack Krupansky
>>>
>>> On Fri, Jan 22, 2016 at 10:29 PM, Kevin Burton 
>>> wrote:
>>>
 I sort of agree.. but we are also considering migrating to hourly
 tables.. and what if the single script doesn't run.

 I like having N nodes make changes like this because in my experience
 that central / single box will usually fail at the wrong time :-/



 On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
 wrote:

> Instead of using ZK, why not solve your concurrency problem by
> removing it?  By that, I mean simply have 1 process that creates all your
> tables instead of creating a race condition intentionally?
>
> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton 
> wrote:
>
>> Not sure if this is a bug or not or kind of a *fuzzy* area.
>>
>> In 2.0 this worked fine.
>>
>> We have a bunch of automated scripts that go through and create
>> tables... one per day.
>>
>> at midnight UTC our entire CQL went offline.. .took down our whole
>> app.  ;-/
>>
>> The resolution was a full CQL shut down and then a drop table to
>> remove the bad tables...
>>
>> pretty sure the issue was with schema disagreement.
>>
>> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT
>> EXISTS only checks locally?
>>
>> My work around is going to be to use zookeeper to create a mutex lock
>> during this operation.
>>
>> Any other things I should avoid?
>>
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux
>> Operations Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>>
>>


 --

 We’re hiring if you know of any awesome Java Devops or Linux Operations
 Engineers!

 Founder/CEO Spinn3r.com
 Location: *San Francisco, CA*
 blog: http://burtonator.wordpress.com
 … or check out my Google+ profile
 


>>>
>>
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San 

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Sebastian Estevez]

CASSANDRA-9424 

All the best,


[image: datastax_logo.png] 

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]







DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Sat, Jan 23, 2016 at 12:22 AM, Jack Krupansky 
wrote:

> I recall that there was some discussion last year about this issue of how
> risky it is to do an automated CREATE TABLE IF NOT EXISTS due to the
> unpredictable amount of time it takes for the table creation to fully
> propagate around the full cluster. I think it was recognized as a real
> problem, but without an immediate solution, so the recommended practice for
> now is to only manually perform the operation (sure, it can be scripted,
> but only under manual control) to assure that the operation completes and
> that only one attempt is made to create the table. I don't recall if there
> was a specific Jira assigned, and the antipattern doc doesn't appear to
> reference this scenario. Maybe a committer can shed some more light.
>
> -- Jack Krupansky
>
> On Fri, Jan 22, 2016 at 10:29 PM, Kevin Burton  wrote:
>
>> I sort of agree.. but we are also considering migrating to hourly
>> tables.. and what if the single script doesn't run.
>>
>> I like having N nodes make changes like this because in my experience
>> that central / single box will usually fail at the wrong time :-/
>>
>>
>>
>> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
>> wrote:
>>
>>> Instead of using ZK, why not solve your concurrency problem by removing
>>> it?  By that, I mean simply have 1 process that creates all your tables
>>> instead of creating a race condition intentionally?
>>>
>>> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton  wrote:
>>>
 Not sure if this is a bug or not or kind of a *fuzzy* area.

 In 2.0 this worked fine.

 We have a bunch of automated scripts that go through and create
 tables... one per day.

 at midnight UTC our entire CQL went offline.. .took down our whole app.
  ;-/

 The resolution was a full CQL shut down and then a drop table to remove
 the bad tables...

 pretty sure the issue was with schema disagreement.

 All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT
 EXISTS only checks locally?

 My work around is going to be to use zookeeper to create a mutex lock
 during this operation.

 Any other things I should avoid?


 --

 We’re hiring if you know of any awesome Java Devops or Linux Operations
 Engineers!

 Founder/CEO Spinn3r.com
 Location: *San Francisco, CA*
 blog: http://burtonator.wordpress.com
 … or check out my Google+ profile
 


>>
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>>
>>
>

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Kevin Burton]

Once the CREATE TABLE returns in cqlsh (or programatically) is it safe to
assume it's on all nodes at that point?

If not I'll have to put in even more logic to handle this case..

On Fri, Jan 22, 2016 at 9:22 PM, Jack Krupansky 
wrote:

> I recall that there was some discussion last year about this issue of how
> risky it is to do an automated CREATE TABLE IF NOT EXISTS due to the
> unpredictable amount of time it takes for the table creation to fully
> propagate around the full cluster. I think it was recognized as a real
> problem, but without an immediate solution, so the recommended practice for
> now is to only manually perform the operation (sure, it can be scripted,
> but only under manual control) to assure that the operation completes and
> that only one attempt is made to create the table. I don't recall if there
> was a specific Jira assigned, and the antipattern doc doesn't appear to
> reference this scenario. Maybe a committer can shed some more light.
>
> -- Jack Krupansky
>
> On Fri, Jan 22, 2016 at 10:29 PM, Kevin Burton  wrote:
>
>> I sort of agree.. but we are also considering migrating to hourly
>> tables.. and what if the single script doesn't run.
>>
>> I like having N nodes make changes like this because in my experience
>> that central / single box will usually fail at the wrong time :-/
>>
>>
>>
>> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
>> wrote:
>>
>>> Instead of using ZK, why not solve your concurrency problem by removing
>>> it?  By that, I mean simply have 1 process that creates all your tables
>>> instead of creating a race condition intentionally?
>>>
>>> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton  wrote:
>>>
 Not sure if this is a bug or not or kind of a *fuzzy* area.

 In 2.0 this worked fine.

 We have a bunch of automated scripts that go through and create
 tables... one per day.

 at midnight UTC our entire CQL went offline.. .took down our whole app.
  ;-/

 The resolution was a full CQL shut down and then a drop table to remove
 the bad tables...

 pretty sure the issue was with schema disagreement.

 All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT
 EXISTS only checks locally?

 My work around is going to be to use zookeeper to create a mutex lock
 during this operation.

 Any other things I should avoid?


 --

 We’re hiring if you know of any awesome Java Devops or Linux Operations
 Engineers!

 Founder/CEO Spinn3r.com
 Location: *San Francisco, CA*
 blog: http://burtonator.wordpress.com
 … or check out my Google+ profile
 


>>
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>>
>>
>


-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Jack Krupansky]

I recall that there was some discussion last year about this issue of how
risky it is to do an automated CREATE TABLE IF NOT EXISTS due to the
unpredictable amount of time it takes for the table creation to fully
propagate around the full cluster. I think it was recognized as a real
problem, but without an immediate solution, so the recommended practice for
now is to only manually perform the operation (sure, it can be scripted,
but only under manual control) to assure that the operation completes and
that only one attempt is made to create the table. I don't recall if there
was a specific Jira assigned, and the antipattern doc doesn't appear to
reference this scenario. Maybe a committer can shed some more light.

-- Jack Krupansky

On Fri, Jan 22, 2016 at 10:29 PM, Kevin Burton  wrote:

> I sort of agree.. but we are also considering migrating to hourly tables..
> and what if the single script doesn't run.
>
> I like having N nodes make changes like this because in my experience that
> central / single box will usually fail at the wrong time :-/
>
>
>
> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad 
> wrote:
>
>> Instead of using ZK, why not solve your concurrency problem by removing
>> it?  By that, I mean simply have 1 process that creates all your tables
>> instead of creating a race condition intentionally?
>>
>> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton  wrote:
>>
>>> Not sure if this is a bug or not or kind of a *fuzzy* area.
>>>
>>> In 2.0 this worked fine.
>>>
>>> We have a bunch of automated scripts that go through and create
>>> tables... one per day.
>>>
>>> at midnight UTC our entire CQL went offline.. .took down our whole app.
>>>  ;-/
>>>
>>> The resolution was a full CQL shut down and then a drop table to remove
>>> the bad tables...
>>>
>>> pretty sure the issue was with schema disagreement.
>>>
>>> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT EXISTS
>>> only checks locally?
>>>
>>> My work around is going to be to use zookeeper to create a mutex lock
>>> during this operation.
>>>
>>> Any other things I should avoid?
>>>
>>>
>>> --
>>>
>>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>>> Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> 
>>>
>>>
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Kevin Burton]

I sort of agree.. but we are also considering migrating to hourly tables..
and what if the single script doesn't run.

I like having N nodes make changes like this because in my experience that
central / single box will usually fail at the wrong time :-/



On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad  wrote:

> Instead of using ZK, why not solve your concurrency problem by removing
> it?  By that, I mean simply have 1 process that creates all your tables
> instead of creating a race condition intentionally?
>
> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton  wrote:
>
>> Not sure if this is a bug or not or kind of a *fuzzy* area.
>>
>> In 2.0 this worked fine.
>>
>> We have a bunch of automated scripts that go through and create tables...
>> one per day.
>>
>> at midnight UTC our entire CQL went offline.. .took down our whole app.
>>  ;-/
>>
>> The resolution was a full CQL shut down and then a drop table to remove
>> the bad tables...
>>
>> pretty sure the issue was with schema disagreement.
>>
>> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT EXISTS
>> only checks locally?
>>
>> My work around is going to be to use zookeeper to create a mutex lock
>> during this operation.
>>
>> Any other things I should avoid?
>>
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>>
>>


-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile

Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....

[Jonathan Haddad]

Instead of using ZK, why not solve your concurrency problem by removing
it?  By that, I mean simply have 1 process that creates all your tables
instead of creating a race condition intentionally?

On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton  wrote:

> Not sure if this is a bug or not or kind of a *fuzzy* area.
>
> In 2.0 this worked fine.
>
> We have a bunch of automated scripts that go through and create tables...
> one per day.
>
> at midnight UTC our entire CQL went offline.. .took down our whole app.
>  ;-/
>
> The resolution was a full CQL shut down and then a drop table to remove
> the bad tables...
>
> pretty sure the issue was with schema disagreement.
>
> All our CREATE TABLE use IF NOT EXISTS but I think the IF NOT EXISTS
> only checks locally?
>
> My work around is going to be to use zookeeper to create a mutex lock
> during this operation.
>
> Any other things I should avoid?
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>