Re: Many pending compactions

2015-02-18 Thread Roni Balthazar
Try repair -pr on all nodes. If after that you still have issues, you can try to rebuild the SSTables using nodetool upgradesstables or scrub. Regards, Roni Balthazar Em 18/02/2015, às 14:13, Ja Sam ptrstp...@gmail.com escreveu: ad 3) I did this already yesterday (setcompactionthrouput

Re: Many pending compactions

2015-02-18 Thread Ja Sam
As Al Tobey suggest me I upgraded my 2.1.0 to snaphot version of 2.1.3. I have now installed exactly this build: https://cassci.datastax.com/job/cassandra-2.1/912/ I see many compaction which completes, but some of them are really slow. Maybe I should send some stats form OpsCenter or servers? But

Re: Many pending compactions

2015-02-18 Thread Jake Luciani
Ja, Please upgrade to official 2.1.3 we've fixed many things related to compaction. Are you seeing the compactions % complete progress at all? On Wed, Feb 18, 2015 at 11:58 AM, Roni Balthazar ronibaltha...@gmail.com wrote: Try repair -pr on all nodes. If after that you still have issues, you

Re: Cassandra install on JRE vs JDK

2015-02-18 Thread Robert Stupp
The ”natural” dependency of Cassandra is the JRE (not the JDK) - e.g. in the Debian package. You should be safe using JRE instead of JDK. If you’re asking whether to use a non-Oracle JVM - the answer would be: use the Oracle JVM. OpenJDK might work, but I’d not recommend it. Am 18.02.2015 um

Re: Deleting Statistics.db at startup

2015-02-18 Thread Robert Coli
On Wed, Feb 18, 2015 at 4:02 AM, Tomer Pearl tomer.pe...@contextream.com wrote: My question is what is the consequences of deleting this file every time the node is starting up? Performance wise or other. You waste the time Cassandra spends to regenerate it. I personally would not institute

C* 2.1.2 invokes oom-killer

2015-02-18 Thread Michał Łowicki
Hi, Couple of times a day 2 out of 4 members cluster nodes are killed root@db4:~# dmesg | grep -i oom [4811135.792657] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [6559049.307293] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Nodes are

Re: C* 2.1.2 invokes oom-killer

2015-02-18 Thread Robert Coli
On Wed, Feb 18, 2015 at 10:28 AM, Michał Łowicki mlowi...@gmail.com wrote: Couple of times a day 2 out of 4 members cluster nodes are killed This sort of issue is usually best handled/debugged interactively on IRC. But briefly : - 2.1.2 is IMO broken for production. Downgrade (officially

Re: Cassandra install on JRE vs JDK

2015-02-18 Thread Mark Reddy
Yes you can use Oracle JDK if your prefer, I've been using the JDK with Cassandra in production for years without issue. Regards, Mark On 18 February 2015 at 19:49, cass savy casss...@gmail.com wrote: Can we install Oracle JDK instead of JRE in Cassandra servers? We have few clusters running

Re: Cassandra install on JRE vs JDK

2015-02-18 Thread cass savy
Thanks Mark for quick response. What version of Cassandra and JDK are you using in Prod. On Wed, Feb 18, 2015 at 11:58 AM, Mark Reddy mark.l.re...@gmail.com wrote: Yes you can use Oracle JDK if your prefer, I've been using the JDK with Cassandra in production for years without issue.

Re: Cassandra install on JRE vs JDK

2015-02-18 Thread cass savy
Thanks Robert for quick response. I use Oracle JDK and not OpenJDK. On Wed, Feb 18, 2015 at 11:54 AM, Robert Stupp sn...@snazy.de wrote: The ”natural” dependency of Cassandra is the JRE (not the JDK) - e.g. in the Debian package. You should be safe using JRE instead of JDK. If you’re

Deleting Statistics.db at startup

2015-02-18 Thread Tomer Pearl
Hello, I have received the following error ERROR [SSTableBatchOpen:2] 2015-01-19 13:55:28,478 CassandraDaemon.java (line 196) Exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.OutOfMemoryError: Java heap space at

Re: Many pending compactions

2015-02-18 Thread Ja Sam
I don't have problems with DC_B (replica) only in DC_A(my system write only to it) I have read timeouts. I checked in OpsCenter SSTable count and I have: 1) in DC_A same +-10% for last week, a small increase for last 24h (it is more than 15000-2 SSTables depends on node) 2) in DC_B last 24h

Re: Adding new node to cluster

2015-02-18 Thread Jonathan Lacefield
Hello, Please note that DataStax has updated the documentation for replacing a seed node. The new docs outline a simplified process to help avoid the confusion on this topic. http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_seed_node.html Jonathan

Re: Many pending compactions

2015-02-18 Thread Ja Sam
Hi, Thanks for your tip it looks that something changed - I still don't know if it is ok. My nodes started to do more compaction, but it looks that some compactions are really slow. In IO we have idle, CPU is quite ok (30%-40%). We set compactionthrouput to 999, but I do not see difference. Can

Re: Many pending compactions

2015-02-18 Thread Ja Sam
1) we tried to run repairs but they usually does not succeed. But we had Leveled compaction before. Last week we ALTER tables to STCS, because guys from DataStax suggest us that we should not use Leveled and alter tables in STCS, because we don't have SSD. After this change we did not run any

Re: Adding new node to cluster

2015-02-18 Thread Batranut Bogdan
Hello, I have decommissioned a node, deleted data,commitlog and saved caches, changed yaml file to not include self ip and started it. For some reason I do not fully understand, Opscenter says that the node is in an unknown datacenter. Nodetool says UJ but shows ? in the Owns column. I have

Re: Many pending compactions

2015-02-18 Thread Roni Balthazar
Are you running repairs within gc_grace_seconds? (default is 10 days) http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_repair_nodes_c.html Double check if you set cold_reads_to_omit to 0.0 on tables with STCS that you do not read often. Are you using default values

Re: Many pending compactions

2015-02-18 Thread Roni Balthazar
You are right... Repair makes the data consistent between nodes. I understand that you have 2 issues going on. You need to run repair periodically without errors and need to decrease the numbers of compactions pending. So I suggest: 1) Run repair -pr on all nodes. If you upgrade to the new

Re: Many pending compactions

2015-02-18 Thread Roni Balthazar
Hi, You can check if the number of SSTables is decreasing. Look for the SSTable count information of your tables using nodetool cfstats. The compaction history can be viewed using nodetool compactionhistory. About the timeouts, check this out:

Re: Many pending compactions

2015-02-18 Thread Roni Balthazar
Which error are you getting when running repairs? You need to run repair on your nodes within gc_grace_seconds (eg: weekly). They have data that are not read frequently. You can run repair -pr on all nodes. Since you do not have deletes, you will not have trouble with that. If you have deletes,

RE: Data tiered compaction and data model question

2015-02-18 Thread Mohammed Guller
What is the maximum number of events that you expect in a day? What is the worst-case scenario? Mohammed From: cass savy [mailto:casss...@gmail.com] Sent: Wednesday, February 18, 2015 4:21 PM To: user@cassandra.apache.org Subject: Data tiered compaction and data model question We want to track

Logging client ID for YCSB workloads on Cassandra?

2015-02-18 Thread Jatin Ganhotra
Hi, I'd like to log the client ID for every operation performed by the YCSB on my Cassandra cluster. The purpose is to identify analyze various other consistency measures other than eventual consistency. I wanted to know if people have done something similar in the past. Or am I missing

Data tiered compaction and data model question

2015-02-18 Thread cass savy
We want to track events in log Cf/table and should be able to query for events that occurred in range of mins or hours for given day. Multiple events can occur in a given minute. Listed 2 table designs and leaning towards table 1 to avoid large wide row. Please advice on *Table 1*: not very

Re: run cassandra on a small instance

2015-02-18 Thread Tim Dunphy
2.1.2 is IMO broken and should not be used for any purpose. Use 2.1.1 or 2.1.3. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob Cool man. Thanks for the info. I just upgraded to 2.1.3. We'll see how that goes. I can let you know more once it's been running

Re: run cassandra on a small instance

2015-02-18 Thread Andrew
Robert, Let me know if I’m off base about this—but I feel like I see a lot of posts that are like this (i.e., use this arbitrary version, not this other arbitrary version).  Why are releases going out if they’re “broken”?  This seems like a very confusing way for new (and existing) users to