> > > - Heap size is set to 8GB > - Using G1GC > - I tried moving the memtable out of the heap. It helped but I still got > an OOM last night > - Concurrent compactors is set to 1 but it still happens and also tried > setting throughput between 16 and 128, no changes. >
That heap size is way to small for G1GC. Switch back to the defaults with CMS. IME, G1 needs > 20g for *just* the JVM to see improvements (but this also depends on workload and a few other factors). Stick with the CMS defaults unless you have some evidence-based experiment to try. Also worth noting that with a 1TB gp2 EBS volume, you only have 3k IOPS to play with before you are subject to rate limiting. If you allocate a volume greater than 3.33TB, you get 10K IOPS and the rate limiting goes away (you can see this playing around with the EBS sizing in the AWS calculator: http://calculator.s3.amazonaws.com/index.html). Another common mistake here is accidentally putting the commitlog on the boot volume which has a super low amount of IOPS given it's 64g (?iirc) by default.
