Some things to consider:
Check for contention around the switch lock. This can happen if you get a lot
of tables flushing at the same time, or if you have a lot of secondary indexes.
It shows up as a pattern in the logs. As soon a the writer starts flushing a
memtable another will be queued.
Oh, and my startup command that cassandra logged was
a2.bigde.nrel.gov: xss = -ea -javaagent:/opt/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8021M -Xmx8021M
-Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k
And I remember from docs you don't want to
