Re: High read latency after data volume increased

2015-01-12 Thread Jonathan Lacefield
There's likely 2 things occurring 1) the cfhistograms error is due to https://issues.apache.org/jira/browse/CASSANDRA-8028 Which is resolved in 2.1.3. Looks like voting is under way for 2.1.3. As rcoli mentioned, you are running the latest open source of C* which should be treated as beta until

Re: High read latency after data volume increased

2015-01-09 Thread Roni Balthazar
Hi there, The compaction remains running with our workload. We are using SATA HDDs RAIDs. When trying to run cfhistograms on our user_data table, we are getting this message: nodetool: Unable to compute when histogram overflowed Please see what happens when running some queries on this cf:

Re: High read latency after data volume increased

2015-01-09 Thread Brian Tarbox
C* seems to have more than its share of version x doesn't work, use version y type issues On Thu, Jan 8, 2015 at 2:23 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar ronibaltha...@gmail.com wrote: We are using C* 2.1.2 with 2 DCs. 30 nodes DC1

Re: High read latency after data volume increased

2015-01-08 Thread Roni Balthazar
Hi Robert, We downgraded to 2.1.1, but got the very same result. The read latency is still high, but we figured out that it happens only using a specific keyspace. Please see the graphs below... ​ Trying another keyspace with 600+ reads/sec, we are getting the acceptable ~30ms read latency. Let

Re: High read latency after data volume increased

2015-01-08 Thread Robert Coli
On Thu, Jan 8, 2015 at 6:38 PM, Roni Balthazar ronibaltha...@gmail.com wrote: We downgraded to 2.1.1, but got the very same result. The read latency is still high, but we figured out that it happens only using a specific keyspace. Note that downgrading is officially unsupported, but is

High read latency after data volume increased

2015-01-08 Thread Roni Balthazar
Hi there, We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. While our data volume is increasing (34 TB now), we are running into some problems: 1) Read latency is around 1000 ms when running 600 reads/sec (DC1 CL.LOCAL_ONE). At the same time the load average is about 20-30 on all

Re: High read latency after data volume increased

2015-01-08 Thread Robert Coli
On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar ronibaltha...@gmail.com wrote: We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ 2.1.2 in particular is known to have significant issues. You'd be better