Re: Cassandra query degradation with high frequency updated tables.

2015-10-10 Thread Brice Dutheil
What do you mean by that *And since this is a test , this is just running on a single node.* ? What is the hardware spec ? Also from the schema in CASSANDRA-10502 there’s a lot of maps, what is the size fo these maps ? I’ve seen cassandra

Re: Cassandra query degradation with high frequency updated tables.

2015-10-09 Thread Nazario Parsacala
So the trace is varying a lot. And does not seem to correlate with the data return from the client ? Maybe datastax java driver related. ..? (not likely).. Just checkout the results. Below is the one that I took when from the client (java application) perspective it was returning data in

Re: Cassandra query degradation with high frequency updated tables.

2015-10-09 Thread Jonathan Haddad
I'd be curious to see GC logs. jstat -gccause On Fri, Oct 9, 2015 at 2:16 PM Tyler Hobbs wrote: > Hmm, it seems off to me that the merge step is taking 1 to 2 seconds, > especially when there are only ~500 cells from one sstable and the > memtables. Can you open a ticket

Re: Cassandra query degradation with high frequency updated tables.

2015-10-09 Thread Nazario Parsacala
I will send the jstat output later. I have created the ticket: https://issues.apache.org/jira/browse/CASSANDRA-10502 > On Oct 9, 2015, at 5:20 PM, Jonathan Haddad wrote: > > I'd be curious to see GC logs. > > jstat -gccause > > On Fri, Oct 9, 2015 at 2:16 PM Tyler

Re: Cassandra query degradation with high frequency updated tables.

2015-10-09 Thread Tyler Hobbs
Hmm, it seems off to me that the merge step is taking 1 to 2 seconds, especially when there are only ~500 cells from one sstable and the memtables. Can you open a ticket ( https://issues.apache.org/jira/browse/CASSANDRA) with your schema, details on your data layout, and these traces? On Fri,

Re: Cassandra query degradation with high frequency updated tables.

2015-10-09 Thread Nazario Parsacala
So I upgraded to 2.2.2 and change the compaction strategy from DateTieredCompactionStrategy to LeveledCompactionStrategy. But the problem still exists. At the start we were getting responses around 80 to a couple of hundred of ms. But after 1.5 hours of running, it is now hitting 1447 ms. I

Re: Cassandra query degradation with high frequency updated tables.

2015-10-09 Thread Tyler Hobbs
That looks like CASSANDRA-10478 , which will probably result in 2.2.3 being released shortly. I'm not sure how that affects performance, but as mentioned in the ticket, you can add "disk_access_mode: standard" to cassandra.yaml to avoid it.

Re: Cassandra query degradation with high frequency updated tables.

2015-10-09 Thread Carlos Alonso
Yeah, I was about to suggest the compaction strategy too. Leveled compaction sounds like a better fit when records are being updated Carlos Alonso | Software Engineer | @calonso On 8 October 2015 at 22:35, Tyler Hobbs wrote: > Upgrade to 2.2.2.

Re: Cassandra query degradation with high frequency updated tables.

2015-10-09 Thread Nazario Parsacala
Compaction did not help too. > On Oct 9, 2015, at 1:01 PM, Nazario Parsacala wrote: > > So I upgraded to 2.2.2 and change the compaction strategy from > DateTieredCompactionStrategy to LeveledCompactionStrategy. But the problem > still exists. > At the start we were

Re: Cassandra query degradation with high frequency updated tables.

2015-10-08 Thread Tyler Hobbs
Upgrade to 2.2.2. Your sstables are probably not compacting due to CASSANDRA-10270 , which was fixed in 2.2.2. Additionally, you may want to look into using leveled compaction (