Jürgen Albersdorfer created CASSANDRA-14239:
-----------------------------------------------
Summary: OutOfMemoryError when bootstrapping with less than 100GB
RAM
Key: CASSANDRA-14239
URL: https://issues.apache.org/jira/browse/CASSANDRA-14239
Project: Cassandra
Issue Type: Bug
Environment: Details of the bootstrapping Node
* ProLiant BL460c G7
* 56GB RAM
* 2x 146GB 10K HDD (One dedicated for Commitlog, one for Data, Hints and
saved_caches)
* CentOS 7.4 on SD-Card
* /tmp and /var/log on tmpfs
* Oracle JDK 1.8.0_151
* Cassandra 3.11.1
Cluster
* 10 existing Nodes (Up and Normal)
Reporter: Jürgen Albersdorfer
Attachments: Objects-by-class.csv,
Objects-with-biggest-retained-size.csv, cassandra-env.sh, cassandra.yaml,
jvm.options, stack-traces.txt
Hi, I face an issue when bootstrapping a Node having less than 100GB RAM on our
10 Node C* 3.11.1 Cluster.
During bootstrap, when I watch the cassandra.log I observe a growth in JVM Heap
Old Gen which gets not significantly freed up any more.
I know that JVM collects on Old Gen only when really needed. I can see
collections, but there is always a remainder which seems to grow forever
without ever getting freed.
After the Node successfully Joined the Cluster, I can remove the extra RAM I
have given it for bootstrapping without any further effect.
It feels like Cassandra will not forget about every single byte streamed over
the Network over time during bootstrapping, - which would be a memory leak and
a major problem, too.
I was able to produce a HeapDumpOnOutOfMemoryError from a 56GB Node (40 GB
assigned JVM Heap). YourKit Profiler shows huge amount of Memory allocated for
org.apache.cassandra.db.Memtable (22 GB)
org.apache.cassandra.db.rows.BufferCell (19 GB) and java.nio.HeapByteBuffer (11
GB)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
