Re: cassandra freezes

2010-02-26 Thread Boris Shulman
What will be the implications of the fact that cassandra can't keep up with the write? Will the memtables be queued in memory until they are flushed? On Thu, Feb 25, 2010 at 4:56 PM, Jonathan Ellis jbel...@gmail.com wrote: Are you swapping?

Re: cassandra freezes

2010-02-25 Thread Jonathan Ellis
The only kind of freeze that makes sense there is your reads are i/o bound and the extra disk activity is killing you. In that case the fix is to add more RAM, or give less to the JVM so the OS can use more for buffer cache. On Thu, Feb 25, 2010 at 8:01 AM, Boris Shulman shulm...@gmail.com

Re: cassandra freezes

2010-02-25 Thread Jonathan Ellis
Then you should check GC timing with -Xverbose:gc option (see: http://wiki.apache.org/cassandra/RunningCassandra for how to modify jvm options) for a correlation. On Thu, Feb 25, 2010 at 8:09 AM, Boris Shulman shulm...@gmail.com wrote: In these tests I perform only write operations, no reads.

Re: cassandra freezes

2010-02-25 Thread Boris Shulman
I don't think it is gc related issue. There is no correlation between gc times and the freeze times. More over I don't see any gc activity that lasts for omre than o.03 sec. But there is a correlation between disk flushing operations. I've noticed that the system freezes each time when my commit

Re: cassandra freezes

2010-02-25 Thread Ted Zlatanov
On Thu, 25 Feb 2010 08:56:25 -0600 Jonathan Ellis jbel...@gmail.com wrote: JE Are you swapping? JE http://spyced.blogspot.com/2010/01/linux-performance-basics.html JE otherwise there's something wrong w/ your vm (?), disk i/o doesn't JE block incoming writes in cassandra If the user has enough

Re: cassandra freezes

2010-02-24 Thread Jonathan Ellis
On Wed, Feb 24, 2010 at 8:46 PM, Santal Li santal...@gmail.com wrote: BTW: Somebody in my team told me, that if the cassandra managed data was too huge( 15x than heap space) , will cause performance issues, is this true? It really has more to do with what your hot data set is, than absolute

Re: cassandra freezes

2010-02-20 Thread Simon Smith
I'm still in the experimentation stage so perhaps forgive this hypothetical question/idea. I am planning to load balance by putting haproxy in front of the cassandra cluster. First of all, is that a bad idea? Secondly, if I have high enough replication and # of nodes, is it possible and a good

Re: cassandra freezes

2010-02-20 Thread Jonathan Ellis
haproxy should be fine. normal GCs aren't a problem, you don't need to worry about that. what is a problem is when you shove more requests into cassandra than it can handle, so it tries to GC to get enough memory to handle that, then you shove even more requests, so it GC's again, and it spirals

Re: cassandra freezes

2010-02-20 Thread Tatu Saloranta
On Fri, Feb 19, 2010 at 7:40 PM, Santal Li santal...@gmail.com wrote: I meet almost same thing as you. When I do some benchmarks write test, some times one Cassandra will freeze and other node will consider it was shutdown and up after 30+ second. I am using 5 node, each node 8G mem for java

Re: cassandra freezes

2010-02-19 Thread Jonathan Ellis
are you using the old deb package? because that had broken gc settings. On Fri, Feb 19, 2010 at 10:40 PM, Santal Li santal...@gmail.com wrote: I meet almost same thing as you. When I do some benchmarks write test, some times one Cassandra will freeze and other node will consider it was

Re: cassandra freezes

2010-02-19 Thread Santal Li
the GC options as bellow: JVM_OPTS= \ -ea \ -Xms2G \ -Xmx8G \ -XX:SurvivorRatio=8 \ -XX:TargetSurvivorRatio=90 \ -XX:+AggressiveOpts \ -XX:+UseParNewGC \ -XX:+UseConcMarkSweepGC \ -XX:+CMSParallelRemarkEnabled \

cassandra freezes

2010-02-16 Thread Boris Shulman
Hello, I'm running some benchmarks on 2 cassandra nodes each running on 8 cores machine with 16G RAM, 10G for Java heap. I've noticed that during benchmarks with numerous writes cassandra just freeze for several minutes (in those benchmarks I'm writing batches of 10 columns with 1K data each for

Re: cassandra freezes

2010-02-16 Thread Tatu Saloranta
On Tue, Feb 16, 2010 at 6:25 AM, Boris Shulman shulm...@gmail.com wrote: Hello, I'm running some benchmarks on 2 cassandra nodes each running on 8 cores machine with 16G RAM, 10G for Java heap. I've noticed that during benchmarks with numerous writes cassandra just freeze for several minutes