RE: Consistency Issues

[Walsh, Stephen]

No such thing as a stupid question☺
I know they exist in some nodes, but if they replicated correctly is a 
different story.
I’m  checking this one now,

Ok, hooked up OpsCenter to see what it was saying,
Out of the 100 keyspaces creted,
9 are missing one CF
2 are missing two CF’s
1 is missing three CF’s

It looks like the replication of the tables did not complete to all nodes?

Looking at each of the 4 nodes at the keyspace with 3 missing CF’s
(via CQLSH_HOST=x.x.x.x cqlsh & “Describe keyspace XXX;”)

Node 1 : has all CF’s
Node 2 : has all CF’s
Node 3 : has all CF’s
Node 4 : has all CF’s


This is indeed very strange….


From: Carlos Alonso [mailto:i...@mrcalonso.com]
Sent: 01 October 2015 12:05
To: user@cassandra.apache.org
Subject: Re: Consistency Issues

And that's a stupid one, I know, but does the column you're trying to access 
actually exist?

Carlos Alonso | Software Engineer | @calonso

On 1 October 2015 at 11:09, Walsh, Stephen 
> wrote:
I did think of that and they are all the same version ☺


From: Carlos Alonso [mailto:i...@mrcalonso.com]
Sent: 01 October 2015 10:11

To: user@cassandra.apache.org
Subject: Re: Consistency Issues

Hi Stephen.

The UnknownColumnFamilyException made me thought of a possible schema 
disagreement in which any of your nodes has a different version and therefore 
you cannot reach quorum?

Can you run nodetool describecluster and see if all nodes have the same schema 
versions?

Cheers!

Carlos Alonso | Software Engineer | @calonso

On 1 October 2015 at 09:49, Walsh, Stephen 
> wrote:
If you’re looking for the clean-up of the old gen in the jvm heap, it doesn’t 
happen.
We have a new gen turning 15 times before its pushed to old gen.
Seems all our data only has a TTL of 10 seconds – very little data is sent to 
the old gen.

Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is our 
issue.


I’m more worried about error messages in the Cassandra log file that state.


UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find 
cfId=cf411b50-6785-11e5-a435-e7be20c92086

and

cassandra OutboundTcpConnection.java:313 - error writing to Connection.



But I really need to understand this best practice that was mentioned (on 
number of CF’s) by Jack Krupansky.
Anyone more information on this?


Many thanks for all your help guys keep it coming ☺
Steve

From: Ricardo Sancho 
[mailto:sancho.rica...@gmail.com]
Sent: 01 October 2015 09:39
To: user@cassandra.apache.org
Subject: RE: Consistency Issues


Can you tell us how much time your gcs are taking?
Do you see any especially long ones?
On 1 Oct 2015 09:37, "Walsh, Stephen" 
> wrote:
There is no load balancer in front of Cassandra,  it’s in front of our 
application.
Everyone seems hung up on this point? But it’s not the root causing of the 
inconsistency issue.

Can anyone verify the best practice for number of CF’s?


From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: 30 September 2015 18:45
To: user@cassandra.apache.org
Subject: Re: Consistency Issues

On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:

We never had these issue with our first run. Its only when we added another 25% 
of writes.

As Jack said, you are probably pushing your GC over a threshold, leading to 
long pause times and inability to meet quorum.

As Sebastian said, you probably shouldn't need a load balancer in front of 
Cassandra.

=Rob

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.
This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your 

Re: Consistency Issues

[Jake Luciani]

Couple things to try.

1. nodetool resetlocalschema on the nodes with missing CFs. This will
refresh the schema on the local node.
2. upgrade to 2.1.9. There are some pretty major issues in 2.1.6 (nothing
specific to this problem but worth upgrading)

Re: Consistency Issues

[Carlos Alonso]

And that's a stupid one, I know, but does the column you're trying to
access actually exist?

Carlos Alonso | Software Engineer | @calonso 

On 1 October 2015 at 11:09, Walsh, Stephen  wrote:

> I did think of that and they are all the same version J
>
>
>
>
>
> *From:* Carlos Alonso [mailto:i...@mrcalonso.com]
> *Sent:* 01 October 2015 10:11
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> Hi Stephen.
>
>
>
> The UnknownColumnFamilyException made me thought of a possible schema
> disagreement in which any of your nodes has a different version and
> therefore you cannot reach quorum?
>
>
>
> Can you run nodetool describecluster and see if all nodes have the same
> schema versions?
>
>
>
> Cheers!
>
>
> Carlos Alonso | Software Engineer | @calonso 
>
>
>
> On 1 October 2015 at 09:49, Walsh, Stephen 
> wrote:
>
> If you’re looking for the clean-up of the old gen in the jvm heap, it
> doesn’t happen.
>
> We have a new gen turning 15 times before its pushed to old gen.
>
> Seems all our data only has a TTL of 10 seconds – very little data is sent
> to the old gen.
>
>
>
> Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is
> our issue.
>
>
>
>
>
> I’m more worried about error messages in the Cassandra log file that state.
>
>
>
>
>
> UnknownColumnFamilyException reading from socket; closing
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=cf411b50-6785-11e5-a435-e7be20c92086
>
>
>
> and
>
>
>
> cassandra OutboundTcpConnection.java:313 - error writing to Connection.
>
>
>
>
>
>
>
> But I really need to understand this best practice that was mentioned (on
> number of CF’s) by Jack Krupansky.
>
> Anyone more information on this?
>
>
>
>
>
> Many thanks for all your help guys keep it coming J
>
> Steve
>
>
>
> *From:* Ricardo Sancho [mailto:sancho.rica...@gmail.com]
> *Sent:* 01 October 2015 09:39
> *To:* user@cassandra.apache.org
> *Subject:* RE: Consistency Issues
>
>
>
> Can you tell us how much time your gcs are taking?
> Do you see any especially long ones?
>
> On 1 Oct 2015 09:37, "Walsh, Stephen"  wrote:
>
> There is no load balancer in front of Cassandra,  it’s in front of our
> application.
>
> Everyone seems hung up on this point? But it’s not the root causing of the
> inconsistency issue.
>
>
>
> Can anyone verify the best practice for number of CF’s?
>
>
>
>
>
> *From:* Robert Coli [mailto:rc...@eventbrite.com]
> *Sent:* 30 September 2015 18:45
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:
>
>
>
> We never had these issue with our first run. Its only when we added
> another 25% of writes.
>
>
>
> As Jack said, you are probably pushing your GC over a threshold, leading
> to long pause times and inability to meet quorum.
>
>
>
> As Sebastian said, you probably shouldn't need a load balancer in front of
> Cassandra.
>
>
>
> =Rob
>
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>

Re: Consistency Issues

[Carlos Alonso]

Which versions of Cassandra and OpsCenter are you using? Because probably
opscenter and your app are using cql and cqlsh is using thrift or vice
versa and that's why depending on where you access from you see different
things?

Carlos Alonso | Software Engineer | @calonso 

On 1 October 2015 at 13:06, Walsh, Stephen  wrote:

> No such thing as a stupid questionJ
>
> I know they exist in some nodes, but if they replicated correctly is a
> different story.
>
> I’m  checking this one now,
>
>
>
> Ok, hooked up OpsCenter to see what it was saying,
>
> Out of the 100 keyspaces creted,
>
> 9 are missing one CF
>
> 2 are missing two CF’s
>
> 1 is missing three CF’s
>
>
>
> It looks like the replication of the tables did not complete to all nodes?
>
>
>
> Looking at each of the 4 nodes at the keyspace with 3 missing CF’s
>
> (via CQLSH_HOST=x.x.x.x cqlsh & “Describe keyspace XXX;”)
>
>
>
> Node 1 : has all CF’s
>
> Node 2 : has all CF’s
>
> Node 3 : has all CF’s
>
> Node 4 : has all CF’s
>
>
>
>
>
> This is indeed very strange….
>
>
>
>
>
> *From:* Carlos Alonso [mailto:i...@mrcalonso.com]
> *Sent:* 01 October 2015 12:05
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> And that's a stupid one, I know, but does the column you're trying to
> access actually exist?
>
>
> Carlos Alonso | Software Engineer | @calonso 
>
>
>
> On 1 October 2015 at 11:09, Walsh, Stephen 
> wrote:
>
> I did think of that and they are all the same version J
>
>
>
>
>
> *From:* Carlos Alonso [mailto:i...@mrcalonso.com]
> *Sent:* 01 October 2015 10:11
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> Hi Stephen.
>
>
>
> The UnknownColumnFamilyException made me thought of a possible schema
> disagreement in which any of your nodes has a different version and
> therefore you cannot reach quorum?
>
>
>
> Can you run nodetool describecluster and see if all nodes have the same
> schema versions?
>
>
>
> Cheers!
>
>
> Carlos Alonso | Software Engineer | @calonso 
>
>
>
> On 1 October 2015 at 09:49, Walsh, Stephen 
> wrote:
>
> If you’re looking for the clean-up of the old gen in the jvm heap, it
> doesn’t happen.
>
> We have a new gen turning 15 times before its pushed to old gen.
>
> Seems all our data only has a TTL of 10 seconds – very little data is sent
> to the old gen.
>
>
>
> Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is
> our issue.
>
>
>
>
>
> I’m more worried about error messages in the Cassandra log file that state.
>
>
>
>
>
> UnknownColumnFamilyException reading from socket; closing
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=cf411b50-6785-11e5-a435-e7be20c92086
>
>
>
> and
>
>
>
> cassandra OutboundTcpConnection.java:313 - error writing to Connection.
>
>
>
>
>
>
>
> But I really need to understand this best practice that was mentioned (on
> number of CF’s) by Jack Krupansky.
>
> Anyone more information on this?
>
>
>
>
>
> Many thanks for all your help guys keep it coming J
>
> Steve
>
>
>
> *From:* Ricardo Sancho [mailto:sancho.rica...@gmail.com]
> *Sent:* 01 October 2015 09:39
> *To:* user@cassandra.apache.org
> *Subject:* RE: Consistency Issues
>
>
>
> Can you tell us how much time your gcs are taking?
> Do you see any especially long ones?
>
> On 1 Oct 2015 09:37, "Walsh, Stephen"  wrote:
>
> There is no load balancer in front of Cassandra,  it’s in front of our
> application.
>
> Everyone seems hung up on this point? But it’s not the root causing of the
> inconsistency issue.
>
>
>
> Can anyone verify the best practice for number of CF’s?
>
>
>
>
>
> *From:* Robert Coli [mailto:rc...@eventbrite.com]
> *Sent:* 30 September 2015 18:45
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:
>
>
>
> We never had these issue with our first run. Its only when we added
> another 25% of writes.
>
>
>
> As Jack said, you are probably pushing your GC over a threshold, leading
> to long pause times and inability to meet quorum.
>
>
>
> As Sebastian said, you probably shouldn't need a load balancer in front of
> Cassandra.
>
>
>
> =Rob
>
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not 

Re: Consistency Issues

[Onur Yalazı]

Thank you Jake.

The issue is I do not have missing CF's and upgrading beyond 2.1.3 is 
not a possibility because of the deprecation of cql dialects. Our 
application is using Hector and migrating to cql3 is a huge refactoring.



On 01/10/15 15:48, Jake Luciani wrote:

Couple things to try.

1. nodetool resetlocalschema on the nodes with missing CFs. This will 
refresh the schema on the local node.
2. upgrade to 2.1.9. There are some pretty major issues in 2.1.6 
(nothing specific to this problem but worth upgrading)

cqlsh strange inconsistency

[Onur Yalazı]

Hello,

I have a table with a status field and a secondary index on that table. 
Every row has 1 or -1 status values.


I have these queries run from cqlsh with the results:

consistency all;
select count(*) from table; /** 1901
Select count(*) from table where status=1; /** 27
select count(*) from table where status=-1; /** 267

This is really strange. If I select status from the table, I see 1901 
rows with status 1 or -1.
I rebuild the index and the results did not change. But If I update 
every row with the correct status value, cassandra starts reporting 
correct counts.


What is happening? How can this situation happen? Do you have any ideas?

Cassandra version: 2.1.3
cqlsh version: 4.1.1

Re: Consistency Issues

[Jake Luciani]

Onur, was responding to Stephen's issue.


On Thu, Oct 1, 2015 at 8:56 AM, Onur Yalazı  wrote:

> Thank you Jake.
>
> The issue is I do not have missing CF's and upgrading beyond 2.1.3 is not
> a possibility because of the deprecation of cql dialects. Our application
> is using Hector and migrating to cql3 is a huge refactoring.
>
>
>
> On 01/10/15 15:48, Jake Luciani wrote:
>
>> Couple things to try.
>>
>> 1. nodetool resetlocalschema on the nodes with missing CFs. This will
>> refresh the schema on the local node.
>> 2. upgrade to 2.1.9. There are some pretty major issues in 2.1.6 (nothing
>> specific to this problem but worth upgrading)
>>
>
>


-- 
http://twitter.com/tjake

RE: Consistency Issues

[Walsh, Stephen]

I did think of that and they are all the same version ☺


From: Carlos Alonso [mailto:i...@mrcalonso.com]
Sent: 01 October 2015 10:11
To: user@cassandra.apache.org
Subject: Re: Consistency Issues

Hi Stephen.

The UnknownColumnFamilyException made me thought of a possible schema 
disagreement in which any of your nodes has a different version and therefore 
you cannot reach quorum?

Can you run nodetool describecluster and see if all nodes have the same schema 
versions?

Cheers!

Carlos Alonso | Software Engineer | @calonso

On 1 October 2015 at 09:49, Walsh, Stephen 
> wrote:
If you’re looking for the clean-up of the old gen in the jvm heap, it doesn’t 
happen.
We have a new gen turning 15 times before its pushed to old gen.
Seems all our data only has a TTL of 10 seconds – very little data is sent to 
the old gen.

Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is our 
issue.


I’m more worried about error messages in the Cassandra log file that state.


UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find 
cfId=cf411b50-6785-11e5-a435-e7be20c92086

and

cassandra OutboundTcpConnection.java:313 - error writing to Connection.



But I really need to understand this best practice that was mentioned (on 
number of CF’s) by Jack Krupansky.
Anyone more information on this?


Many thanks for all your help guys keep it coming ☺
Steve

From: Ricardo Sancho 
[mailto:sancho.rica...@gmail.com]
Sent: 01 October 2015 09:39
To: user@cassandra.apache.org
Subject: RE: Consistency Issues


Can you tell us how much time your gcs are taking?
Do you see any especially long ones?
On 1 Oct 2015 09:37, "Walsh, Stephen" 
> wrote:
There is no load balancer in front of Cassandra,  it’s in front of our 
application.
Everyone seems hung up on this point? But it’s not the root causing of the 
inconsistency issue.

Can anyone verify the best practice for number of CF’s?


From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: 30 September 2015 18:45
To: user@cassandra.apache.org
Subject: Re: Consistency Issues

On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:

We never had these issue with our first run. Its only when we added another 25% 
of writes.

As Jack said, you are probably pushing your GC over a threshold, leading to 
long pause times and inability to meet quorum.

As Sebastian said, you probably shouldn't need a load balancer in front of 
Cassandra.

=Rob

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.
This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

RE: Consistency Issues

[Walsh, Stephen]

Thanks Jake, I’ll try test out 2.1.9 to see if it resolved the issue and ill 
try “nodetool resetlocalschema” now to see if it helps.

Cassandra is 2.1.6
OpsCenter is 5.2.1

From: Jake Luciani [mailto:jak...@gmail.com]
Sent: 01 October 2015 14:00
To: user 
Subject: Re: Consistency Issues

Onur, was responding to Stephen's issue.


On Thu, Oct 1, 2015 at 8:56 AM, Onur Yalazı 
> wrote:
Thank you Jake.

The issue is I do not have missing CF's and upgrading beyond 2.1.3 is not a 
possibility because of the deprecation of cql dialects. Our application is 
using Hector and migrating to cql3 is a huge refactoring.



On 01/10/15 15:48, Jake Luciani wrote:
Couple things to try.

1. nodetool resetlocalschema on the nodes with missing CFs. This will refresh 
the schema on the local node.
2. upgrade to 2.1.9. There are some pretty major issues in 2.1.6 (nothing 
specific to this problem but worth upgrading)




--
http://twitter.com/tjake
This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

Re: Fwd: Column family ID mismatch

[Sebastian Estevez]

Check this post & my response:

http://stackoverflow.com/questions/31576180/cassandra-2-1-system-schema-missing=0CBoQFjAAahUKEwit6uK1uqHIAhXLcD4KHWsxDK8=AFQjCNFfvPld2OOInC6B_M0X_QpfqHUbxA=aLeCqqe7vhpQlL10p9WFJA
On Oct 1, 2015 4:33 AM, "kedar"  wrote:

> Got this error again on a single node cassandra.
>
> Would appreciate some pointers.
>
>  Forwarded Message  Subject: Column family ID mismatch Date:
> Thu, 13 Aug 2015 20:44:04 +0530 From: kedar 
>  Reply-To: user@cassandra.apache.org To:
> user@cassandra.apache.org 
> 
>
> Hi All,
>
> My keyspace is created as:
>
> CREATE KEYSPACE  WITH replication = {'class':
> 'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = true;
>
> However I am running a single node cluster:
>
> ./nodetool status 
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address Load   Tokens  Owns (effective)  Host
> ID   Rack
> XX  XXX.XXX.XXX.XX  3.73 GB256 100.0%  rack1
>
> And things were still running fine till today we encountered:
>
> ERROR [MigrationStage:1] 2015-08-13 01:58:49,249 CassandraDaemon.java:153
> - Exception in thread Thread[MigrationStage:1,5,main]
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found ; expected )
> at
> org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1125)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:393)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column
> family ID mismatch (found ; expected )
> at
> org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1208)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1140)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1121)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ... 11 common frames omitted
>
>
> Did notetool repair, probably didn't work so after few mins did a restart
> and then the problem went away.
>
> Need help in understanding what caused it and how it was resolved.
>
> Thanks
>
>
>
>

Re: Consistency Issues

[Jonathan Haddad]

You say that you don't think GC is your issue... but did you actually
check?  The reasons you suggest aren't very convincing.  Can you provide
your GC settings, and take a look at jstat --gccause?

http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstat.html#gccause_option



On Thu, Oct 1, 2015 at 4:50 AM Walsh, Stephen 
wrote:

> If you’re looking for the clean-up of the old gen in the jvm heap, it
> doesn’t happen.
>
> We have a new gen turning 15 times before its pushed to old gen.
>
> Seems all our data only has a TTL of 10 seconds – very little data is sent
> to the old gen.
>
>
>
> Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is
> our issue.
>
>
>
>
>
> I’m more worried about error messages in the Cassandra log file that state.
>
>
>
>
>
> UnknownColumnFamilyException reading from socket; closing
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=cf411b50-6785-11e5-a435-e7be20c92086
>
>
>
> and
>
>
>
> cassandra OutboundTcpConnection.java:313 - error writing to Connection.
>
>
>
>
>
>
>
> But I really need to understand this best practice that was mentioned (on
> number of CF’s) by Jack Krupansky.
>
> Anyone more information on this?
>
>
>
>
>
> Many thanks for all your help guys keep it coming J
>
> Steve
>
>
>
> *From:* Ricardo Sancho [mailto:sancho.rica...@gmail.com]
> *Sent:* 01 October 2015 09:39
> *To:* user@cassandra.apache.org
> *Subject:* RE: Consistency Issues
>
>
>
> Can you tell us how much time your gcs are taking?
> Do you see any especially long ones?
>
> On 1 Oct 2015 09:37, "Walsh, Stephen"  wrote:
>
> There is no load balancer in front of Cassandra,  it’s in front of our
> application.
>
> Everyone seems hung up on this point? But it’s not the root causing of the
> inconsistency issue.
>
>
>
> Can anyone verify the best practice for number of CF’s?
>
>
>
>
>
> *From:* Robert Coli [mailto:rc...@eventbrite.com]
> *Sent:* 30 September 2015 18:45
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:
>
>
>
> We never had these issue with our first run. Its only when we added
> another 25% of writes.
>
>
>
> As Jack said, you are probably pushing your GC over a threshold, leading
> to long pause times and inability to meet quorum.
>
>
>
> As Sebastian said, you probably shouldn't need a load balancer in front of
> Cassandra.
>
>
>
> =Rob
>
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>

JSON Order By

[Ashish Soni]

Hi All ,
I have a question related to JSON sorting
I have a below structure stored in cassandra and i would like to get the
internal array sorted by a property when i select it , Please let me know
if there is way to do that .

I need to sort the rules Array by property ruleOrder when i select

CREATE TYPE IF NOT EXISTS Rule (  condition text,  action text,  ruleOrder
int);

INSERT INTO model.RuleSetSchedule JSON ' {
ruleSetName": "RATES",
"ruleSetId": "829aa84b-4bba-411f-a4fb-38167a987cda",
"scheduleId":1,
"effectiveStartDate": "2015-02-01 00:00:00",
"effectiveEndDate": "2015-03-01 00:00:00",
"rules": {
"1": {
"condition": "BoardStation ==''Lowell''",
"action": "FareAmount=9.25",
   * "ruleOrder": 2*
}
 "2": {
"condition": "BoardStation ==''Lowell''",
"action": "FareAmount=9.25",
   * "ruleOrder": 1*
}

}
}';

Re: Consistency Issues

[Carlos Alonso]

Well... I wasn't expecting that, as both OpsCenter 5.2.1 and cqlsh in
Cassandra 2.1.x both use native protocol. I was expecting them having
different protocols, so

Have no further ideas :(

Carlos Alonso | Software Engineer | @calonso 

On 1 October 2015 at 14:36, Walsh, Stephen  wrote:

> Thanks Jake, I’ll try test out 2.1.9 to see if it resolved the issue and
> ill try “nodetool resetlocalschema” now to see if it helps.
>
>
>
> Cassandra is 2.1.6
>
> OpsCenter is 5.2.1
>
>
>
> *From:* Jake Luciani [mailto:jak...@gmail.com]
> *Sent:* 01 October 2015 14:00
> *To:* user 
> *Subject:* Re: Consistency Issues
>
>
>
> Onur, was responding to Stephen's issue.
>
>
>
>
>
> On Thu, Oct 1, 2015 at 8:56 AM, Onur Yalazı 
> wrote:
>
> Thank you Jake.
>
> The issue is I do not have missing CF's and upgrading beyond 2.1.3 is not
> a possibility because of the deprecation of cql dialects. Our application
> is using Hector and migrating to cql3 is a huge refactoring.
>
>
>
>
> On 01/10/15 15:48, Jake Luciani wrote:
>
> Couple things to try.
>
> 1. nodetool resetlocalschema on the nodes with missing CFs. This will
> refresh the schema on the local node.
> 2. upgrade to 2.1.9. There are some pretty major issues in 2.1.6 (nothing
> specific to this problem but worth upgrading)
>
>
>
>
>
>
>
> --
>
> http://twitter.com/tjake
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>

RE: Consistency Issues

[Sebastian Estevez]

You're running describe with CL quorum aren't you?

To see the inconsistency you'd have to check the system.schema_column
family tables on each node.
On Oct 1, 2015 8:07 AM, "Walsh, Stephen"  wrote:

> No such thing as a stupid questionJ
>
> I know they exist in some nodes, but if they replicated correctly is a
> different story.
>
> I’m  checking this one now,
>
>
>
> Ok, hooked up OpsCenter to see what it was saying,
>
> Out of the 100 keyspaces creted,
>
> 9 are missing one CF
>
> 2 are missing two CF’s
>
> 1 is missing three CF’s
>
>
>
> It looks like the replication of the tables did not complete to all nodes?
>
>
>
> Looking at each of the 4 nodes at the keyspace with 3 missing CF’s
>
> (via CQLSH_HOST=x.x.x.x cqlsh & “Describe keyspace XXX;”)
>
>
>
> Node 1 : has all CF’s
>
> Node 2 : has all CF’s
>
> Node 3 : has all CF’s
>
> Node 4 : has all CF’s
>
>
>
>
>
> This is indeed very strange….
>
>
>
>
>
> *From:* Carlos Alonso [mailto:i...@mrcalonso.com]
> *Sent:* 01 October 2015 12:05
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> And that's a stupid one, I know, but does the column you're trying to
> access actually exist?
>
>
> Carlos Alonso | Software Engineer | @calonso 
>
>
>
> On 1 October 2015 at 11:09, Walsh, Stephen 
> wrote:
>
> I did think of that and they are all the same version J
>
>
>
>
>
> *From:* Carlos Alonso [mailto:i...@mrcalonso.com]
> *Sent:* 01 October 2015 10:11
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> Hi Stephen.
>
>
>
> The UnknownColumnFamilyException made me thought of a possible schema
> disagreement in which any of your nodes has a different version and
> therefore you cannot reach quorum?
>
>
>
> Can you run nodetool describecluster and see if all nodes have the same
> schema versions?
>
>
>
> Cheers!
>
>
> Carlos Alonso | Software Engineer | @calonso 
>
>
>
> On 1 October 2015 at 09:49, Walsh, Stephen 
> wrote:
>
> If you’re looking for the clean-up of the old gen in the jvm heap, it
> doesn’t happen.
>
> We have a new gen turning 15 times before its pushed to old gen.
>
> Seems all our data only has a TTL of 10 seconds – very little data is sent
> to the old gen.
>
>
>
> Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is
> our issue.
>
>
>
>
>
> I’m more worried about error messages in the Cassandra log file that state.
>
>
>
>
>
> UnknownColumnFamilyException reading from socket; closing
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=cf411b50-6785-11e5-a435-e7be20c92086
>
>
>
> and
>
>
>
> cassandra OutboundTcpConnection.java:313 - error writing to Connection.
>
>
>
>
>
>
>
> But I really need to understand this best practice that was mentioned (on
> number of CF’s) by Jack Krupansky.
>
> Anyone more information on this?
>
>
>
>
>
> Many thanks for all your help guys keep it coming J
>
> Steve
>
>
>
> *From:* Ricardo Sancho [mailto:sancho.rica...@gmail.com]
> *Sent:* 01 October 2015 09:39
> *To:* user@cassandra.apache.org
> *Subject:* RE: Consistency Issues
>
>
>
> Can you tell us how much time your gcs are taking?
> Do you see any especially long ones?
>
> On 1 Oct 2015 09:37, "Walsh, Stephen"  wrote:
>
> There is no load balancer in front of Cassandra,  it’s in front of our
> application.
>
> Everyone seems hung up on this point? But it’s not the root causing of the
> inconsistency issue.
>
>
>
> Can anyone verify the best practice for number of CF’s?
>
>
>
>
>
> *From:* Robert Coli [mailto:rc...@eventbrite.com]
> *Sent:* 30 September 2015 18:45
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:
>
>
>
> We never had these issue with our first run. Its only when we added
> another 25% of writes.
>
>
>
> As Jack said, you are probably pushing your GC over a threshold, leading
> to long pause times and inability to meet quorum.
>
>
>
> As Sebastian said, you probably shouldn't need a load balancer in front of
> Cassandra.
>
>
>
> =Rob
>
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose 

Cassandra certification

[Fernandez Gomara, Ruben (CCI-Atlanta)]

Hi,
Did anybody took the Cassandra (Administrator) certification on the Summit? Any 
feedback? How was the test?
Thanks
Ruben

Re: Cassandra certification

[Tushar Agrawal]

Check this out:
Get Trained, Get Certified, Get Better Paid

https://www.linkedin.com/pulse/get-trained-certified-better-paid-tushar-agrawal

Thanks,
Tushar
(Sent from iPhone)

> On Oct 1, 2015, at 9:00 AM, Fernandez Gomara, Ruben (CCI-Atlanta) 
>  wrote:
> 
> Hi,
> Did anybody took the Cassandra (Administrator) certification on the Summit? 
> Any feedback? How was the test?
> Thanks
> Ruben
>  

Re: memory usage problem of Metadata.tokenMap.tokenToHost

[Alexandre Dutra]

I confirm that the optional schema metadata feature (JAVA-828
) has been merged
upstream and is included in driver 3.0.0-alpha3 and will as well be
included in the forthcoming 2.1.8 version.

On a side note, JAVA-664
 has been open for
quite a while now and aims to fix the problem you describe. It will
hopefully make it into 2.1.8 as well, although I cannot guarantee that at
this point.

Thanks,

Alexandre Dutra

On Mon, Sep 28, 2015 at 7:55 PM Alex Popescu  wrote:

> Besides the others' advice that 2000+ keyspaces might be too much, the
> latest Java driver (2.0.11) includes an option to disable the Metadata API
> http://www.datastax.com/dev/blog/datastax-java-driver-2-0-11-released.
> I'm not sure at this moment if this has been merged into 2.1 already.
>
> On Sun, Sep 20, 2015 at 9:22 AM, joseph gao 
> wrote:
>
>> cassandra: 2.1.7
>> java driver: datastax java driver 2.1.6
>>
>> Here is the problem:
>>My application uses 2000+ keyspaces, and will dynamically create
>> keyspaces and tables. And then in java client, the
>> Metadata.tokenMap.tokenToHost would use about 1g memory. so this will cause
>> a lot of  full gc.
>>As I see, the key of the tokenToHost is keyspace, and the value is a
>> tokenId_to_replicateNodes map.
>>
>>When I try to solve this problem, I find something not sure: all
>> keyspaces have same 'tokenId_to_replicateNodes' map.
>> My replication strategy of all keyspaces is : simpleStrategy and
>> replicationFactor is 3
>>
>> So would it be possible if keyspaces use same strategy, the value of
>> tokenToHost map use a same map. So it would extremely reduce the memory
>> usage
>>
>>  thanks a lot
>>
>> --
>> --
>> Joseph Gao
>> PhoneNum:15210513582
>> QQ: 409343351
>>
>
>
>
> --
> Bests,
>
> Alex Popescu | @al3xandru
> Sen. Product Manager @ DataStax
>
>
>
> 
>
-- 
Alexandre Dutra
Driver & Tools Engineer @ DataStax

Re: cqlsh strange inconsistency

[Robert Coli]

On Thu, Oct 1, 2015 at 5:37 AM, Onur Yalazı  wrote:

> What is happening? How can this situation happen? Do you have any ideas?

 ...

>  This is really strange. If I select status from the table, I see 1901
> rows with status 1 or -1.

I rebuild the index and the results did not change. But If I update every
> row with the correct status value, cassandra starts reporting correct
> counts.


There is an endless trickle of such issues with secondary indexes. This is
one of the reasons I do not personally recommend their use.

If you have a repro test case, file it at issues.apache.org and let the
list know the issue URL?

=Rob

RE: Cassandra certification

[SEAN_R_DURITY]

I took (and passed) the Admin exam at the Cassandra Summit. Tushar does hit the 
main points in his article. One of the conditions of the test is to not reveal 
details on the test, so we shouldn’t reveal anything very substantial. I will 
say that the exam was about the same difficulty level as Informix certification 
that I took several years ago. It is not simple. You have to know the material 
AND be careful about the test’s format.

Sean Durity – Lead Cassandra Admin
Big DATA Team

From: Tushar Agrawal [mailto:agrawal.tus...@gmail.com]
Sent: Thursday, October 01, 2015 12:01 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra certification

Check this out:
Get Trained, Get Certified, Get Better Paid

https://www.linkedin.com/pulse/get-trained-certified-better-paid-tushar-agrawal

Thanks,
Tushar
(Sent from iPhone)

On Oct 1, 2015, at 9:00 AM, Fernandez Gomara, Ruben (CCI-Atlanta) 
> wrote:
Hi,
Did anybody took the Cassandra (Administrator) certification on the Summit? Any 
feedback? How was the test?
Thanks
Ruben




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

GossipingPropertyFileSnitch and nodetool info

[Eric Plowe]

I am using C* 2.1.9 and GossipingPropertyFileSnitch. I noticed that when I
run nodetool info I am seeing the data center and rack as

DC1
r1

which comes from the default entry from cassandra-topology.properties. From
my understanding, this should only be used when PropertyFileSnitch is used
and as a fallback when using GossipingPropertyFileSnitch

When I run nodetool status, and looking at the cluster in ops center,
everything reports correctly.

As a test, I renamed cassandra-topology.properties to
cassandra-topology.properties.back on a node and restarted C*. When I run
nodetool info I am getting:

Data Center: UNKNOWN_DC
Rack   : UNKNOWN_RACK

nodetool status and opscenter still report correctly.

Is this just a bug with nodetool?

Regards,

Eric Plowe

Re: broadcast address on EC2 without Elastic IPs.

[Bryan Cheng]

Hey Renato,

As far as I can tell, the reason you're getting private IP addresses back
is that the node you're connecting to is relaying back the way that _it_
knows where to find other nodes, which is a function of the gossip state.
This is expected behavior.

Mixed Private/Public IP spaces without full connectivity between both sets
of address spaces is always going to be a pain, IMHO; you're much better
off standardizing on one or the other.

It sounds like you have a machine (maybe a dev machine?) outside of EC2
trying to reach your cluster. If this is just for development, then the
"easiest" mechanism would be to standardize on public IPs; elastic IP's are
free as long as they're in use, so I would just request a quota increase.

If this is a production configuration (maybe another datacenter?) you'll
probably want to investigate a more robust routing solution. We have two
datacenters with distinct, non-intersecting Private IP spaces; we use a VPN
to route between them. Clients on both sides of the tunnel can natively
speak the private IP's on the other side, which eliminates odd issues from
NAT. From what I've seen on the list, this is a somewhat common
configuration.

Hope this helps!

--Bryan

On Wed, Sep 30, 2015 at 7:24 AM, Renato Perini 
wrote:

> Hello!
> I have configured a small cluster composed of three nodes on Amazon EC2.
> The 3 machines don't have an elastic IP (static address) so the public
> address changes at every reboot.
>
> I have a machine with a static ip that I use as a bridge to access the
> other 3 cassandra nodes through SSH. On this machine, I have setup a
> tunnelling towards the first node of the cluster in order to open the 9042
> port and let me access the cluster through this static IP.
>
> Basically, my cassandra.yaml has these settings:
> listen_address: private IP
> broadcast_address: commented out.
> rpc_address: 0.0.0.0
> broadcast_rpc_address: private ip
>
> I know I should set the broadcast address to the public IP, but it is
> dynamic and I don't have any idea at the moment on how I could determine it
> and setup it in the cassandra.yaml file.
>
> I'm developing a small client using the datastax connector (in Java).
> I setup the contactpoint using the public ip of the bridge machine. The
> client connects but gives some errors while adding other nodes in the
> cluster:
>
> 15:43:26,887 ERROR [com.datastax.driver.core.Session]
> (cluster1-nio-worker-1) Error creating pool to /XXX.XX.XX.XXX:9042:
> com.datastax.driver.core.TransportException: [/XXX.XX.XX.XXX:9042] Cannot
> connect
> at
> com.datastax.driver.core.Connection$1.operationComplete(Connection.java:156)
> [cassandra-driver-core-2.2.0-rc3.jar:]
> at
> com.datastax.driver.core.Connection$1.operationComplete(Connection.java:139)
> [cassandra-driver-core-2.2.0-rc3.jar:]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:214)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_80]
> Caused by: io.netty.channel.ConnectTimeoutException: connection timed out:
> /XXX.XX.XX.XXX:9042
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:212)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> ... 6 more
>
> 15:43:26,887 ERROR [com.datastax.driver.core.Session]
> (cluster1-nio-worker-3) Error creating pool to /XXX.XX.XX.XX:9042:
> com.datastax.driver.core.TransportException: [/XXX.XX.XX.XX:9042] Cannot
> connect
> at
> com.datastax.driver.core.Connection$1.operationComplete(Connection.java:156)
> [cassandra-driver-core-2.2.0-rc3.jar:]
> at
> 

Re: addition of nodes with auth enabled on a datacenter causes existing nodes to loose their permissions

[Graham Sanderson]

You are seeing 

https://issues.apache.org/jira/browse/CASSANDRA-9519 


> On Oct 1, 2015, at 9:16 PM, K F  wrote:
> 
> Hi,
> 
> I have 3 DCs out of which in one of the DC, I added 20 nodes. All of the DCs 
> had auth enabled, it was functioning fine. But after addition of 20 nodes in 
> one of the DC, the permissions just got messed-up on the existing nodes. My 
> application started getting errors while querying using the user it normally 
> did. 
> 
> Finally, I logged onto one of the existing nodes that was operating fine and 
> issued the following cql query and it gave me the following error.
> 
> cqlsh:system_auth> select * from users;
> TSocket read 0 bytes
> 
> Upon investigation in system log I found the following exception, what does 
> this mean? Thanks.
> 
> 2015-10-02 02:03:10,229 [RPC-Thread:3] ERROR Message Unexpected throwable 
> while invoking!
> java.lang.IllegalArgumentException: Comparison method violates its general 
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:868)
> at java.util.TimSort.mergeAt(TimSort.java:485)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
> at java.util.TimSort.sort(TimSort.java:223)
> at java.util.TimSort.sort(TimSort.java:173)
> at java.util.Arrays.sort(Arrays.java:659)
> at java.util.Collections.sort(Collections.java:217)
> at 
> org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49)
> at 
> org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157)
> at 
> org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186)
> at 
> org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151)
> at 
> org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1483)
> at 
> org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:1545)
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:258)
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:228)
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:64)
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158)
> at 
> com.datastax.bdp.cassandra.cql3.DseQueryHandler$StatementExecution.execute(DseQueryHandler.java:448)
> at 
> com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithTiming(DseQueryHandler.java:190)
> at 
> com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithAuditLogging(DseQueryHandler.java:223)
> at 
> com.datastax.bdp.cassandra.cql3.DseQueryHandler.process(DseQueryHandler.java:103)
> at 
> org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1958)
> at 
> com.datastax.bdp.server.DseServer.execute_cql3_query(DseServer.java:543)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4486)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4470)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>   
> 



smime.p7s
Description: S/MIME cryptographic signature

addition of nodes with auth enabled on a datacenter causes existing nodes to loose their permissions

[K F]

Hi,
I have 3 DCs out of which in one of the DC, I added 20 nodes. All of the DCs 
had auth enabled, it was functioning fine. But after addition of 20 nodes in 
one of the DC, the permissions just got messed-up on the existing nodes. My 
application started getting errors while querying using the user it normally 
did. 
Finally, I logged onto one of the existing nodes that was operating fine and 
issued the following cql query and it gave me the following error.
cqlsh:system_auth> select * from users;TSocket read 0 bytes
Upon investigation in system log I found the following exception, what does 
this mean? Thanks.
2015-10-02 02:03:10,229 [RPC-Thread:3] ERROR Message Unexpected throwable while 
invoking!java.lang.IllegalArgumentException: Comparison method violates its 
general contract!        at java.util.TimSort.mergeHi(TimSort.java:868)        
at java.util.TimSort.mergeAt(TimSort.java:485)        at 
java.util.TimSort.mergeForceCollapse(TimSort.java:426)        at 
java.util.TimSort.sort(TimSort.java:223)        at 
java.util.TimSort.sort(TimSort.java:173)        at 
java.util.Arrays.sort(Arrays.java:659)        at 
java.util.Collections.sort(Collections.java:217)        at 
org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49)
        at 
org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157)
        at 
org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186)
        at 
org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151)
        at 
org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1483)
        at 
org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:1545) 
       at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:258)
        at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:228)
        at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:64)
        at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158)
        at 
com.datastax.bdp.cassandra.cql3.DseQueryHandler$StatementExecution.execute(DseQueryHandler.java:448)
        at 
com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithTiming(DseQueryHandler.java:190)
        at 
com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithAuditLogging(DseQueryHandler.java:223)
        at 
com.datastax.bdp.cassandra.cql3.DseQueryHandler.process(DseQueryHandler.java:103)
        at 
org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1958)
        at 
com.datastax.bdp.server.DseServer.execute_cql3_query(DseServer.java:543)        
at 
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4486)
        at 
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4470)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)   
     at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)        
at com.thinkaurelius.thrift.Message.invoke(Message.java:314)        at 
com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)        at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
        at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)        
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
       at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
       at java.lang.Thread.run(Thread.java:745)  

Re: Cassandra users survey

[Jim Ancona]

Hi Jonathan,

The survey asks about "your application." We have multiple applications
using Cassandra. Are you looking for information about each application
separately, or the sum of all of them?

Jim

On Wed, Sep 30, 2015 at 2:18 PM, Jonathan Ellis  wrote:

> With 3.0 approaching, the Apache Cassandra team would appreciate your
> feedback as we work on the project roadmap for future releases.
>
> I've put together a brief survey here:
> https://docs.google.com/forms/d/1TEG0umQAmiH3RXjNYdzNrKoBCl1x7zurMroMzAFeG2Y/viewform?usp=send_form
>
> Please take a few minutes to fill it out!
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
>
>

Fwd: Column family ID mismatch

[kedar]

Got this error again on a single node cassandra.

Would appreciate some pointers.

 Forwarded Message 
Subject:Column family ID mismatch
Date:   Thu, 13 Aug 2015 20:44:04 +0530
From:   kedar 
Reply-To:   user@cassandra.apache.org
To: user@cassandra.apache.org 



Hi All,

My keyspace is created as:

CREATE KEYSPACE  WITH replication = {'class': 
'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = true;


However I am running a single node cluster:

./nodetool status 
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address Load   Tokens  Owns (effective)  Host 
ID   Rack

XX  XXX.XXX.XXX.XX  3.73 GB256 100.0%   rack1

And things were still running fine till today we encountered:

ERROR [MigrationStage:1] 2015-08-13 01:58:49,249 
CassandraDaemon.java:153 - Exception in thread 
Thread[MigrationStage:1,5,main]
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ConfigurationException: Column family ID 
mismatch (found ; expected )
at 
org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1125) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:393) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
~[na:1.7.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
~[na:1.7.0_65]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
~[na:1.7.0_65]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_65]

at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Caused by: org.apache.cassandra.exceptions.ConfigurationException: 
Column family ID mismatch (found ; expected )
at 
org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1208) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1140) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1121) 
~[apache-cassandra-2.1.2.jar:2.1.2]

... 11 common frames omitted


Did notetool repair, probably didn't work so after few mins did a 
restart and then the problem went away.


Need help in understanding what caused it and how it was resolved.

Thanks

RE: Consistency Issues

[Walsh, Stephen]

There is no load balancer in front of Cassandra,  it’s in front of our 
application.
Everyone seems hung up on this point? But it’s not the root causing of the 
inconsistency issue.

Can anyone verify the best practice for number of CF’s?


From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: 30 September 2015 18:45
To: user@cassandra.apache.org
Subject: Re: Consistency Issues

On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:

We never had these issue with our first run. Its only when we added another 25% 
of writes.

As Jack said, you are probably pushing your GC over a threshold, leading to 
long pause times and inability to meet quorum.

As Sebastian said, you probably shouldn't need a load balancer in front of 
Cassandra.

=Rob

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

RE: Consistency Issues

[Walsh, Stephen]

If you’re looking for the clean-up of the old gen in the jvm heap, it doesn’t 
happen.
We have a new gen turning 15 times before its pushed to old gen.
Seems all our data only has a TTL of 10 seconds – very little data is sent to 
the old gen.

Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is our 
issue.


I’m more worried about error messages in the Cassandra log file that state.


UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find 
cfId=cf411b50-6785-11e5-a435-e7be20c92086

and

cassandra OutboundTcpConnection.java:313 - error writing to Connection.



But I really need to understand this best practice that was mentioned (on 
number of CF’s) by Jack Krupansky.
Anyone more information on this?


Many thanks for all your help guys keep it coming ☺
Steve

From: Ricardo Sancho [mailto:sancho.rica...@gmail.com]
Sent: 01 October 2015 09:39
To: user@cassandra.apache.org
Subject: RE: Consistency Issues


Can you tell us how much time your gcs are taking?
Do you see any especially long ones?
On 1 Oct 2015 09:37, "Walsh, Stephen" 
> wrote:
There is no load balancer in front of Cassandra,  it’s in front of our 
application.
Everyone seems hung up on this point? But it’s not the root causing of the 
inconsistency issue.

Can anyone verify the best practice for number of CF’s?


From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: 30 September 2015 18:45
To: user@cassandra.apache.org
Subject: Re: Consistency Issues

On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:

We never had these issue with our first run. Its only when we added another 25% 
of writes.

As Jack said, you are probably pushing your GC over a threshold, leading to 
long pause times and inability to meet quorum.

As Sebastian said, you probably shouldn't need a load balancer in front of 
Cassandra.

=Rob

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.
This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

Re: Consistency Issues

[Carlos Alonso]

Hi Stephen.

The UnknownColumnFamilyException made me thought of a possible schema
disagreement in which any of your nodes has a different version and
therefore you cannot reach quorum?

Can you run nodetool describecluster and see if all nodes have the same
schema versions?

Cheers!

Carlos Alonso | Software Engineer | @calonso 

On 1 October 2015 at 09:49, Walsh, Stephen  wrote:

> If you’re looking for the clean-up of the old gen in the jvm heap, it
> doesn’t happen.
>
> We have a new gen turning 15 times before its pushed to old gen.
>
> Seems all our data only has a TTL of 10 seconds – very little data is sent
> to the old gen.
>
>
>
> Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is
> our issue.
>
>
>
>
>
> I’m more worried about error messages in the Cassandra log file that state.
>
>
>
>
>
> UnknownColumnFamilyException reading from socket; closing
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=cf411b50-6785-11e5-a435-e7be20c92086
>
>
>
> and
>
>
>
> cassandra OutboundTcpConnection.java:313 - error writing to Connection.
>
>
>
>
>
>
>
> But I really need to understand this best practice that was mentioned (on
> number of CF’s) by Jack Krupansky.
>
> Anyone more information on this?
>
>
>
>
>
> Many thanks for all your help guys keep it coming J
>
> Steve
>
>
>
> *From:* Ricardo Sancho [mailto:sancho.rica...@gmail.com]
> *Sent:* 01 October 2015 09:39
> *To:* user@cassandra.apache.org
> *Subject:* RE: Consistency Issues
>
>
>
> Can you tell us how much time your gcs are taking?
> Do you see any especially long ones?
>
> On 1 Oct 2015 09:37, "Walsh, Stephen"  wrote:
>
> There is no load balancer in front of Cassandra,  it’s in front of our
> application.
>
> Everyone seems hung up on this point? But it’s not the root causing of the
> inconsistency issue.
>
>
>
> Can anyone verify the best practice for number of CF’s?
>
>
>
>
>
> *From:* Robert Coli [mailto:rc...@eventbrite.com]
> *Sent:* 30 September 2015 18:45
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:
>
>
>
> We never had these issue with our first run. Its only when we added
> another 25% of writes.
>
>
>
> As Jack said, you are probably pushing your GC over a threshold, leading
> to long pause times and inability to meet quorum.
>
>
>
> As Sebastian said, you probably shouldn't need a load balancer in front of
> Cassandra.
>
>
>
> =Rob
>
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>

RE: Consistency Issues

[Ricardo Sancho]

Can you tell us how much time your gcs are taking?
Do you see any especially long ones?
On 1 Oct 2015 09:37, "Walsh, Stephen"  wrote:

> There is no load balancer in front of Cassandra,  it’s in front of our
> application.
>
> Everyone seems hung up on this point? But it’s not the root causing of the
> inconsistency issue.
>
>
>
> Can anyone verify the best practice for number of CF’s?
>
>
>
>
>
> *From:* Robert Coli [mailto:rc...@eventbrite.com]
> *Sent:* 30 September 2015 18:45
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen 
> wrote:
>
>
>
> We never had these issue with our first run. Its only when we added
> another 25% of writes.
>
>
>
> As Jack said, you are probably pushing your GC over a threshold, leading
> to long pause times and inability to meet quorum.
>
>
>
> As Sebastian said, you probably shouldn't need a load balancer in front of
> Cassandra.
>
>
>
> =Rob
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>