1. Why 24GB of heap? Do you need this high heap? Bigger heap can lead to longer GC cycles but 15min look too long. 2. Do you have ROW cache enabled? 3. How many column families do you have? 4. Enable GC logs and monitor what GC is doing to get idea of why it is taking so long. You can add following to enable gc log. # GC logging options -- uncomment to enable # JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails" # JVM_OPTS="$JVM_OPTS -XX:+PrintGCTimeStamps" # JVM_OPTS="$JVM_OPTS -XX:+PrintClassHistogram" # JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution" # JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime" # JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc.log"
5. Move to Cassandra 0.7.2, if possible. It has following nice feature: "added flush_largest_memtables_at and reduce_cache_sizes_at options to cassandra.yaml as an escape value for memory pressure" Thanks, Naren On Thu, Feb 24, 2011 at 2:21 PM, Jeffrey Wang <jw...@palantir.com> wrote: > Hey all, > > > > Our setup is 5 machines running Cassandra 0.7.0 with 24GB of heap and 1.5TB > disk each collocated in a DC. We’re doing bulk imports from each of the > nodes with RF = 2 and write consistency ANY (write perf is very important). > The behavior we’re seeing is this: > > > > - Nodes often see each other as dead even though none of the > nodes actually go down. I suspect this may be due to long GCs. It seems like > increasing the RPC timeout could help this, but I’m not convinced this is > the root of the problem. Note that in this case writes return with the > UnavailableException. > > - As mentioned, long GCs. We see the ParNew GC doing a lot of > smaller collections (few hundred MB) which are very fast (few hundred ms), > but every once in a while the ConcurrentMarkSweep will take a LONG time (up > to 15 min!) to collect upwards of 15GB at once. > > - On some nodes, we see a lot of pending MutationStages build up > (e.g. 500K), which leads to the messages “Dropped X MUTATION messages in the > last 5000ms,” presumably meaning that Cassandra has decided to not write one > of the replicas of the data. This is not a HUGE deal, but is less than > ideal. > > - The end result is that a bunch of writes end up failing due to > the UnavailableExceptions, so not all of our data is getting into Cassandra. > > > > So my question is: what is the best way to avoid this behavior? Our > memtable thresholds are fairly low (256MB) so there should be plenty of heap > space to work with. We may experiment with write consistency ONE or ALL to > see if the perf hit is not too bad, but I wanted to get some opinions on why > this might be happening. Thanks! > > > > -Jeffrey > > >
