Host ID collision making node disappear

I have a 3 node Cassandra Cluster. Using DataStax Enetrprise v4.5.1 on VMWare. I am adding 1 new node to this cluster for running Analytics workload. So I cloned existing one of the Cassandra VMs, changed the hostname, restarted VMs, then updated Cassandra.yaml file and restarted Cassandra.

Re: Host ID collision making node disappear

Rahul, I'm pretty sure it's preferable to clean all files and directories in /var/log/cassandra before starting up the new Cassandra node. This will make it start on a clean slate resetting all state from previous node. Cheers, Jens — Sent from Mailbox On Fri, Aug 8, 2014 at 6:21 PM,

Re: Delete By Partition Key Implementation

This is a good question.. I'd love to find out the answer. Seems like a tombstone with prefixes for the keys would work well. Also, can't any key prefixes work in theory? On Thu, Aug 7, 2014 at 8:33 AM, DuyHai Doan doanduy...@gmail.com wrote: Hello all Usually, when using DELETE in CQL3

too many open files

Hi, I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having too many open files exceptions when I try to perform a large number of operations in my 10 node cluster. I saw the documentation

Is per-table memory overhead due to SSTables or tables?

The conventional wisdom says that it's ideal to only use in the low hundreds in the number of tables with cassandra as each table can use 1MB or so of heap. So if you have 1000 tables you'd have 1GB of heap used (which is no fun). But is this an issue with the tables themselves or the SSTables?

Re: too many open files

Are you using apache or Datastax cassandra? The datastax distribution ups the file handle limit to 10. That number's hard to exceed. On Fri, Aug 8, 2014 at 1:35 PM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br wrote: Hi, I am using Cassandra 2.0.9 running on Debian Wheezy, and I am

Re: too many open files

I am using datastax community, the packaged version for Debian. I am also using last version of opscenter and datastax-agent However, I just listed open files here: sudo lsof -n | grep java | wc -l 1096599 It seems it has exceed. Should I just increase? Or is it possible to be a memory leak?

Re: too many open files

You may want to look at the the actual filenames. You might have an app leaving them open. Also, remember, sockets use FDs so they are in the list too. On Fri, Aug 8, 2014 at 1:13 PM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br wrote: I am using datastax community, the packaged version

nodetool cleanup error on 2.0.3

After joining a node to an existing cluster, I run 'nodetool cleanup' as recommended on existing nodes in the cluster. On one node, after some time, I am getting an error starting with Exception in thread main java.lang.AssertionError:

Strange slow schema agreement on 2.0.9 ... anyone seen this?

We recently upgraded C* from 2.0.5 to 2.0.9 We have some data that is partitioned in tables created periodically (once a day). This morning, this automated process timed out because the schema did not reach agreement quickly enough after we created a new empty table. I was able to reproduce

Re: Strange slow schema agreement on 2.0.9 ... anyone seen this?

On Fri, Aug 8, 2014 at 1:45 PM, graham sanderson gra...@vast.com wrote: We have some data that is partitioned in tables created periodically (once a day). This morning, this automated process timed out because the schema did not reach agreement quickly enough after we created a new empty

spark on cassandra

HI, are there any cluster specific prerequisites for running spark on Cassandra? I create two DCs. DC1 and DC2. DC1 had two cassandra nodes with vnodes. I create two nodes in DC2 with murmu partitioning and set num_token: 1. Enabled Hadoop and Spark and started DSE. I can verify that hadoop

Re: Delete By Partition Key Implementation

A deletion of an entire row is a single row tombstone, and yes there are range tombstones for marking deletion of a range of columns also On Aug 8, 2014, at 2:17 PM, Kevin Burton bur...@spinn3r.com wrote: This is a good question.. I'd love to find out the answer. Seems like a tombstone with

Re: Strange slow schema agreement on 2.0.9 ... anyone seen this?

Ok thanks - I guess I can at least enable the debug logging added for that issue to see if it is deliberately choosing not to pull the schema… no repro case, but it may happen again! On Aug 8, 2014, at 4:21 PM, Robert Coli rc...@eventbrite.com wrote: On Fri, Aug 8, 2014 at 1:45 PM, graham

Re: too many open files

I just solved the issue, it was Cassandra process which was opening too many fds, indeed, but the problem was the amount of client connections being opened. It was opening more connection than needed in the client' side. Thanks for the help. []s 2014-08-08 17:17 GMT-03:00 Kevin Burton

Re: Strange slow schema agreement on 2.0.9 ... anyone seen this?

Actually I think it is a different issue (or a freak issue)… the invocation in InternalResponseStage is part of the “schema pull” mechanism this ticket relates to, and in my case this is actually repairing (thank you) the schema disagreement because as a result of it eventually being noticed by

Re: too many open files

You may have this problem if your client doesn't reuse the connection but opens new every type. So, run netstat and check the number of established connections. This number should not be big. Thank you, Andrey On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br

Re: too many open files

Just to chime in, I also ran into this issue when I was migrating to the Datastax client. Instead of reusing the session, I was opening a new session each time. For some reason, even though I was still closing the session on the client side, I was getting the same error. Plus, the only way I

Re: too many open files

On Fri, Aug 8, 2014 at 5:52 PM, Redmumba redmu...@gmail.com wrote: Just to chime in, I also ran into this issue when I was migrating to the Datastax client. Instead of reusing the session, I was opening a new session each time. For some reason, even though I was still closing the session on

Re: Is per-table memory overhead due to SSTables or tables?

See https://issues.apache.org/jira/browse/CASSANDRA-5935 2.1 has a radically different implementation that side steps this (with off heap memtables), but if you really want lots of tables now you can do so as a trade off against GC behavior. The problem is not SSTables per se, but more

Re: Is per-table memory overhead due to SSTables or tables?

hm.. as a side note, it's amazing how much cassandra information is locked up in JIRAs… wonder if there's a way to compute automatically the JIRAs with important information. On Fri, Aug 8, 2014 at 5:14 PM, graham sanderson gra...@vast.com wrote: See

Re: Is per-table memory overhead due to SSTables or tables?

google ;-) On Aug 8, 2014, at 7:33 PM, Kevin Burton bur...@spinn3r.com wrote: hm.. as a side note, it's amazing how much cassandra information is locked up in JIRAs… wonder if there's a way to compute automatically the JIRAs with important information. On Fri, Aug 8, 2014 at 5:14 PM,

Re: too many open files

Yes, definitely look how many open files are actual file handles versus networks sockets. We found a file handle leak in 2.0 but it was patched in 2.0.3 or .5 I think. A million open files is way too high. On Fri, Aug 8, 2014 at 5:19 PM, Andrey Ilinykh ailin...@gmail.com wrote: You may have

Re: too many open files

For cassandra driver,session is just like database connection pool,it maybe contains many tcp connections,if you create a new session every time,more and more tcp connections will be created,till surpass the max file description limit of os. You should create one session,use it repeatedly

Re: too many open files

Indeed, that was my mistake, that was exactly what we were doing in the code. []s 2014-08-09 0:56 GMT-03:00 Brian Zhang yikebo...@gmail.com: For cassandra driver,session is just like database connection pool,it maybe contains many tcp connections,if you create a new session every time,more