Re: High read latency after data volume increased

2015-01-08 Thread Roni Balthazar
Hi Robert, We downgraded to 2.1.1, but got the very same result. The read latency is still high, but we figured out that it happens only using a specific keyspace. Please see the graphs below... ​ Trying another keyspace with 600+ reads/sec, we are getting the acceptable ~30ms read latency. Let

C* throws OOM error despite use of automatic paging

2015-01-08 Thread Mohammed Guller
Hi - We have an ETL application that reads all rows from Cassandra (2.1.2), filters them and stores a small subset in an RDBMS. Our application is using Datastax's Java driver (2.1.4) to fetch data from the C* nodes. Since the Java driver supports automatic paging, I was under the impression

Re: High read latency after data volume increased

2015-01-08 Thread Robert Coli
On Thu, Jan 8, 2015 at 6:38 PM, Roni Balthazar ronibaltha...@gmail.com wrote: We downgraded to 2.1.1, but got the very same result. The read latency is still high, but we figured out that it happens only using a specific keyspace. Note that downgrading is officially unsupported, but is

Re: Rebooted cassandra node timing out all requests but recovers after a while

2015-01-08 Thread Anand Somani
I will keep an eye for that if it happens again. Times at this point are synchronized On Wed, Jan 7, 2015 at 10:37 PM, Duncan Sands duncan.sa...@gmail.com wrote: Hi Anand, On 08/01/15 02:02, Anand Somani wrote: Hi, We have a 3 node cluster (on VM). Eg. host1, host2, host3. One of the VM

User audit in Cassandra

2015-01-08 Thread Ajay
Hi, Is there a way to enable user audit or trace if we have enabled PasswordAuthenticator in cassandra.yaml and set up the users as well. I noticed there are keyspaces system_auth and system_trace. But there is no way to find out which user initiated which session. Is there anyway to find out?.

Why does C* repeatedly compact the same tables over and over?

2015-01-08 Thread Robert Wille
After bootstrapping a node, the node repeatedly compacts the same tables over and over, even though my cluster is completely idle. I’ve noticed the same behavior after extended periods of heavy writes. I realize that during bootstrapping (or extended periods of heavy writes) that compaction

incremental repairs

2015-01-08 Thread Roland Etzenhammer
Hi, I am currently trying to migrate my test cluster to incremental repairs. These are the steps I'm doing on every node: - touch marker - nodetool disableautocompation - nodetool repair - cassandra stop - find all *Data*.db files older then marker - invoke sstablerepairedset on those -

Re: incremental repairs

2015-01-08 Thread Marcus Eriksson
If you are on 2.1.2+ (or using STCS) you don't those steps (should probably update the blog post). Now we keep separate levelings for the repaired/unrepaired data and move the sstables over after the first incremental repair But, if you are running 2.1 in production, I would recommend that you

Re: incremental repairs

2015-01-08 Thread Roland Etzenhammer
Hi Marcus, thanks for that quick reply. I did also look at: http://www.datastax.com/documentation/cassandra/2.1/cassandra/operations/ops_repair_nodes_c.html which describes the same process, it's 2.1.x, so I see that 2.1.2+ is not covered there. I did upgrade my testcluster to 2.1.2 and with

Re: incremental repairs

2015-01-08 Thread Marcus Eriksson
Yes, you should reenable autocompaction /Marcus On Thu, Jan 8, 2015 at 10:33 AM, Roland Etzenhammer r.etzenham...@t-online.de wrote: Hi Marcus, thanks for that quick reply. I did also look at: http://www.datastax.com/documentation/cassandra/2.1/

Re: Why does C* repeatedly compact the same tables over and over?

2015-01-08 Thread Eric Stevens
Are you using Leveled compaction strategy? If you fall behind on compaction in leveled (and you will during bootstrap), by default Cassandra will fall back to size tiered compaction in level 0. This will cause SSTables larger than sstable_size_in_mb, and those will be recompacted away into level

Re: How to bulkload into a specific data center?

2015-01-08 Thread Ryan Svihla
Just noticed you'd sent this to the dev list, this is a question for only the user list, and please do not send questions of this type to the developer list. On Thu, Jan 8, 2015 at 8:33 AM, Ryan Svihla r...@foundev.pro wrote: The nature of replication factor is such that writes will go wherever

Re: Why does C* repeatedly compact the same tables over and over?

2015-01-08 Thread mck
Are you using Leveled compaction strategy? And if you're using Date Tiered compaction strategy on a table that isn't time-series data, for example deletes happen, you find it compacting over and over. ~mck

Re: How to bulkload into a specific data center?

2015-01-08 Thread Ryan Svihla
The nature of replication factor is such that writes will go wherever there is replication. If you're wanting responses to be faster, and not involve the REST data center in the spark job for response I suggest using a cql driver and LOCAL_ONE or LOCAL_QUORUM consistency level (look at the spark

Updated JMX metrics overview

2015-01-08 Thread Reik Schatz
I am adding some jmx metrics to our monitoring tool. Is there a good overview of all existing jmx metrics and their description? http://wiki.apache.org/cassandra/Metrics has only a few metrics. In particular I have questions about these now: org.apache.cassandra.db type=StorageProxy TotalHints -

Re: incremental repairs

2015-01-08 Thread Roland Etzenhammer
Hi Marcus, thanks a lot for those pointers. Now further testing can begin - and I'll wait for 2.1.3. Right now on production repair times are really painful, maybe that will become better. At least I hope so :-)

Re: incremental repairs

2015-01-08 Thread Robert Coli
On Thu, Jan 8, 2015 at 12:28 AM, Marcus Eriksson krum...@gmail.com wrote: But, if you are running 2.1 in production, I would recommend that you wait until 2.1.3 is out, https://issues.apache.org/jira/browse/CASSANDRA-8316 fixes a bunch of issues with incremental repairs There are other

High read latency after data volume increased

2015-01-08 Thread Roni Balthazar
Hi there, We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. While our data volume is increasing (34 TB now), we are running into some problems: 1) Read latency is around 1000 ms when running 600 reads/sec (DC1 CL.LOCAL_ONE). At the same time the load average is about 20-30 on all

Re: High read latency after data volume increased

2015-01-08 Thread Robert Coli
On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar ronibaltha...@gmail.com wrote: We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ 2.1.2 in particular is known to have significant issues. You'd be better