16GB heap is too small for G1GC . Try at least 32GB of heap size On Tue, Aug 22, 2017 at 7:58 AM Fay Hou [Storage Service] < fay...@coupang.com> wrote:
> What errors do you see? > 16gb of 256 GB . Heap is too small. I would give heap at least 160gb. > > > On Aug 22, 2017 7:42 AM, "Thakrar, Jayesh" <jthak...@conversantmedia.com> > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > We are somewhat new users to Cassandra 3.10 on Linux and wanted to ping > the user group for their experiences. > > > > > > Our usage profile is batch jobs that load millions of rows to Cassandra > every hour. > > > And there are similar period batch jobs that read millions of rows and do > some processing, outputting the result to HDFS (no issues with HDFS). > > > > > > We often seen Cassandra daemons crash. > > > Key points of our environment are: > > > *Pretty good servers:* 54 cores (with hyperthreading), 256 GB RAM, 3.2 TB > SSD drive > > > *Compaction:* TWCS compaction with 7 day windows as the data retention > period is limited - about 120 days. > > > *JDK: *Java 1.8.0.71 and G1 GC > > > > *Heap Size:* 16 GB > > > *Large SSTables:* 50 GB to 300+ GB > > > > > > > > We see the daemons crash after some back-to-back long GCs (1.5 to 3.5 > seconds). > > > Note that we had set the target for GC pauses to be 200 ms > > > > > > We have been somewhat able to tame the crashes by updating the TWCS > compaction properties > > > > to have min/max compaction sstables = 4 and by drastically reducing the > size of the New/Eden space (to 5% of heap space = 800 MB). > > > Its been about 12 hours and our stop-the-world gc pauses are under 90 ms. > > > Since the servers have more than sufficient resources, we are not seeing > any noticeable performance impact. > > > > > > Is this kind of tuning normal/expected? > > > > > > Thanks, > > > Jayesh > > > > > > > > > > > > >
