[
https://issues.apache.org/jira/browse/CASSANDRA-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526919#comment-14526919
]
Anuj commented on CASSANDRA-8150:
---------------------------------
We have write heavy workload and used to face promotion failures/long gc pauses
with Cassandra 2.0.x. I am not into code yet but I think that memtable and
compaction related objects have mid-life and write heavy workload is not
suitable for generation collection by default. So, we tuned JVM to make sure
that minimum objects are promoted to Old Gen and achieved great success in that:
MAX_HEAP_SIZE="12G"
HEAP_NEWSIZE="3G"
-XX:SurvivorRatio=2
-XX:MaxTenuringThreshold=20
-XX:CMSInitiatingOccupancyFraction=70
JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=20"
JVM_OPTS="$JVM_OPTS -XX:+UnlockDiagnosticVMOptions"
JVM_OPTS="$JVM_OPTS -XX:+UseGCTaskAffinity"
JVM_OPTS="$JVM_OPTS -XX:+BindGCTaskThreadsToCPUs"
JVM_OPTS="$JVM_OPTS -XX:ParGCCardsPerStrideChunk=32768"
JVM_OPTS="$JVM_OPTS -XX:+CMSScavengeBeforeRemark"
JVM_OPTS="$JVM_OPTS -XX:CMSMaxAbortablePrecleanTime=30000"
JVM_OPTS="$JVM_OPTS -XX:CMSWaitDuration=2000"
JVM_OPTS="$JVM_OPTS -XX:+CMSEdenChunksRecordAlways"
JVM_OPTS="$JVM_OPTS -XX:+CMSParallelInitialMarkEnabled"
JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"
We also think that default total_space_in_mb=1/4 heap is too much for write
heavy loads. By default, young gen is also 1/4 heap.We reduced it to 1000mb in
order to make sure that memtable related objects dont stay in memory for too
long. Combining this with SurvivorRatio=2 and MaxTenuringThreshold=20 did the
job well. GC was very consistent. No Full GC observed.
Environment: 3 node cluster with each node having 24cores,64G RAM and SSDs in
RAID5.
We are making around 12k writes/sec in 5 cf (one with 4 sec index) and 2300
reads/sec on each node of 3 node cluster. 2 CFs have wide rows with max data of
around 100mb per row
> Revaluate Default JVM tuning parameters
> ---------------------------------------
>
> Key: CASSANDRA-8150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8150
> Project: Cassandra
> Issue Type: Improvement
> Components: Config
> Reporter: Matt Stump
> Assignee: Ryan McGuire
> Attachments: upload.png
>
>
> It's been found that the old twitter recommendations of 100m per core up to
> 800m is harmful and should no longer be used.
> Instead the formula used should be 1/3 or 1/4 max heap with a max of 2G. 1/3
> or 1/4 is debatable and I'm open to suggestions. If I were to hazard a guess
> 1/3 is probably better for releases greater than 2.1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
