[ 
https://issues.apache.org/jira/browse/CASSANDRA-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434042#comment-16434042
 ] 

Sergey Kirillov edited comment on CASSANDRA-14239 at 4/11/18 3:11 PM:
----------------------------------------------------------------------

Ok. Now it stuck on 5134 pending MemtableFlushWriter jobs. Number is not 
decreasing anymore. 

*UPD* When everything was blocked node had high CPU usage and where reading a 
lot from disks (which seems related to CASSANDRA-13065).
After a while number of pending memtable jobs decreased and mutations 
unblocked, but in a minute node died again with OOM.


was (Author: rushman):
Ok. Now it stuck on 5134 pending MemtableFlushWriter jobs. Number is not 
decreasing anymore. 

> OutOfMemoryError when bootstrapping with less than 100GB RAM
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-14239
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14239
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Details of the bootstrapping Node
>  * ProLiant BL460c G7
>  * 56GB RAM
>  * 2x 146GB 10K HDD (One dedicated for Commitlog, one for Data, Hints and 
> saved_caches)
>  * CentOS 7.4 on SD-Card
>  * /tmp and /var/log on tmpfs
>  * Oracle JDK 1.8.0_151
>  * Cassandra 3.11.1
> Cluster
>  * 10 existing Nodes (Up and Normal)
>            Reporter: J├╝rgen Albersdorfer
>            Priority: Major
>              Labels: materializedviews
>         Attachments: Objects-by-class.csv, 
> Objects-with-biggest-retained-size.csv, Selection_420.png, Selection_421.png, 
> cassandra-env.sh, cassandra.yaml, dstat.png, gc.log.0.201804111524.zip, 
> gc.log.0.current.zip, gc.log.201804111141.zip, jvm.options, jvm_opts.txt, 
> stack-traces.txt
>
>
> Hi, I face an issue when bootstrapping a Node having less than 100GB RAM on 
> our 10 Node C* 3.11.1 Cluster.
> During bootstrap, when I watch the cassandra.log I observe a growth in JVM 
> Heap Old Gen which gets not significantly freed up any more.
> I know that JVM collects on Old Gen only when really needed. I can see 
> collections, but there is always a remainder which seems to grow forever 
> without ever getting freed.
> After the Node successfully Joined the Cluster, I can remove the extra RAM I 
> have given it for bootstrapping without any further effect.
> It feels like Cassandra will not forget about every single byte streamed over 
> the Network over time during bootstrapping, - which would be a memory leak 
> and a major problem, too.
> I was able to produce a HeapDumpOnOutOfMemoryError from a 56GB Node (40 GB 
> assigned JVM Heap). YourKit Profiler shows huge amount of Memory allocated 
> for org.apache.cassandra.db.Memtable (22 GB) 
> org.apache.cassandra.db.rows.BufferCell (19 GB) and java.nio.HeapByteBuffer 
> (11 GB)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to