cassandra performance problems

2013-12-05 Thread Alexander Shutyaev
Hi all, We have a 3 node cluster setup, single keyspace, about 500 tables. The hardware is 2 cores + 16 GB RAM (Cassandra chose to have 4GB). Cassandra version is 2.0.3. Our replication factor is 3, read/write consistency is QUORUM. We've plugged it into our production environment as a cache in

Replacing a Node using a Replication

2013-12-05 Thread Philippe Dupont
Hi, We currently have a 28 node C* cluster on m1.XLarge instances using Vnodes and are encountering a Raid issue with one of them. The first solution could be to decommission this node and insert a new one in the cluster, since we use vnodes we need to run 28 cleanup after adding a node, this

Re: Raid Issue on EC2 Datastax ami, 1.2.11

2013-12-05 Thread Philippe Dupont
Hi again, I have much more in formations on this case : We did further investigations on the nodes affected and did find some await problems on one of the 4 disk in raid: http://imageshack.com/a/img824/2391/s7q3.jpg Here was the iostat of the node :

Counters question - is there a better way to count

2013-12-05 Thread Christopher Wirt
I want to build a really simple column family which counts the occurrence of a single event X. Once we reach Y occurrences of X the counter resets to 0 The obvious way to do this is with a counter CF. CREATE TABLE xcounter1 ( id uuid, someid int,

Re: Counters question - is there a better way to count

2013-12-05 Thread Andy Twigg
How many distinct uid,someid pairs will you have? On Dec 5, 2013 3:44 PM, Christopher Wirt chris.w...@struq.com wrote: I want to build a really simple column family which counts the occurrence of a single event X. Once we reach Y occurrences of X the counter resets to 0 The obvious way

Re: Counters question - is there a better way to count

2013-12-05 Thread Przemek Maciolek
Some big systems using Cassandra's counters were built (such as Rainbird: http://www.slideshare.net/kevinweil/rainbird-realtime-analytics-at-twitter-strata-2011) and seem to be doing great job. If you are concerned with performance, then maybe using memory-based store (such as Redis) will better

Re: Replacing a Node using a Replication

2013-12-05 Thread Andre Sprenger
We just migrated a Cassandra cluster on EC2 to another instance type. We replaced one server after another, this creates problems similar to what you describe. We simply stop Cassandra, copy the complete data dir to an EBS volume, terminate the server, launch another server with the same IP,

Re: cassandra performance problems

2013-12-05 Thread Jonathan Haddad
Do you mean high CPU usage or high load avg? (20 indicates load avg to me). High load avg means the CPU is waiting on something. Check iostat -dmx 1 100 to check your disk stats, you'll see the columns that indicate mb/s read write as well as % utilization. Once you understand the bottleneck

RE: Counters question - is there a better way to count

2013-12-05 Thread Christopher Wirt
Hi Andy, There will be 10's millions of uid each with 100's of someid being accessed each day. Hi Przemek, We currently use counter column families, but they are some of our slowest. (they are also some of our biggest, so the counter type might not be the issue) We have a strong need

how to find nodes by row key?

2013-12-05 Thread Daneel Yaitskov
Hi, I study cassandra and want to play with it. I need to shrink the number of available row instances. Next send a search request for it in cqlsh with different consistency levels. To observe behavior I need to know which instance to take down. I found cassandra-cli command list. this

Re: how to find nodes by row key?

2013-12-05 Thread Robert Coli
On Thu, Dec 5, 2013 at 9:58 AM, Daneel Yaitskov rtfm.rtfm.r...@gmail.comwrote: The best solution would be get IP address (node UUID) directly by a human readable structured row key. nodetool getendpoints exposes a JMX endpoint which does that. I'm not clear on whether that is sufficient for

Re: cassandra performance problems

2013-12-05 Thread Robert Coli
On Thu, Dec 5, 2013 at 4:33 AM, Alexander Shutyaev shuty...@gmail.comwrote: Cassandra version is 2.0.3. ... We've plugged it into our production environment as a cache in front of postgres. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ What can be the reason?

Drop keyspace via CQL hanging on master/trunk.

2013-12-05 Thread Brian O'Neill
When running Cassandra from trunk/master, I see a drop keyspace command hang at the CQL prompt. To reproduce: 1) Removed my cassandra data directory entirely 2) Fired up cqlsh, and executed the following CQL commands in succession: bone@zen:~/git/boneill42/cassandra- bin/cqlsh Connected to Test

new project - Under Siege

2013-12-05 Thread Jonathan Haddad
I've recently pushed up a new project to github, which we've named Under Siege. It's a java agent for reporting Cassandra metrics to statsd. We've in the process of deploying it to our production clusters. Tested against Cassandra 1.2.11. The metrics library seems to change on every release of

Re: Replacing a Node using a Replication

2013-12-05 Thread Robert Coli
On Thu, Dec 5, 2013 at 8:31 AM, Andre Sprenger andre.spren...@getanet.dewrote: We just migrated a Cassandra cluster on EC2 to another instance type. We replaced one server after another, this creates problems similar to what you describe. We simply stop Cassandra, copy the complete data dir

Re: Notes and questions from performing a large delete

2013-12-05 Thread Nate McCall
Per the 256mb to 5mb change, check the very last section of this page: http://www.datastax.com/documentation/cql/3.0/webhelp/cql/cql_reference/alter_table_r.html Changing any compaction or compression option erases all previous compaction or compression settings. In other words, you have to

vnodes on aws

2013-12-05 Thread Andrey Ilinykh
Hello everybody! We run cassandra 1.1 on ec2 instances. We use three availability zones, the replication factor is 3 also. NetworkTopologyStrategy guarantees each row is replicated in all availability zones. So, if we lost one zone quorum operations still work. We think about to upgrade to 1.2.

Re: Drop keyspace via CQL hanging on master/trunk.

2013-12-05 Thread Jason Wee
Hey Brian, just out of curiosity, why would you remove cassandra data directory entirely? /Jason On Fri, Dec 6, 2013 at 2:38 AM, Brian O'Neill b...@alumni.brown.edu wrote: When running Cassandra from trunk/master, I see a drop keyspace command hang at the CQL prompt. To reproduce: 1)

Re: Drop keyspace via CQL hanging on master/trunk.

2013-12-05 Thread Brian O'Neill
I removed the data directory just to make sure I had a clean environment. (eliminating the possibility of corrupt keyspaces/files causing problems) -brian --- Brian O'Neill Chief Architect Health Market Science The Science of Better Results 2700 Horizon Drive € King of Prussia, PA € 19406 M:

Re: vnodes on aws

2013-12-05 Thread Jayadev Jayaraman
Availability zones are analogous to racks not data centres . EC2 regions are equivalent to data centres. You can use vnodes if you want to deploy a cluster across multiple regions (data centres) with one availability zone per region. Each region maintains a separate ring. I don't know if you can

Re: cassandra performance problems

2013-12-05 Thread Alexander Shutyaev
Thanks for your answers, Jonathan, yes it was load avg and iowait was lower than 2% all that time - the only load was the user one. Robert, we had -Xmx4012m which was automatically calculated by the default cassandra-env.sh (1/4 of total memory - 16G) - we didn't change that. 2013/12/5 Robert