[
https://issues.apache.org/jira/browse/CASSANDRA-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433933#comment-16433933
]
Paulo Motta commented on CASSANDRA-14239:
-----------------------------------------
{quote}Removing MVs is not easy in my case, but now at least I know that it is
worth the efforts.
{quote}
We've done a few improvements to MV bootstrap performance on CASSANDRA-13299
and CASSANDRA-13065, but unfortunately these are only available on 4.0.
The OOMs during bootstrap reported here could probably benefit from
CASSANDRA-13299, so I think it's worth a backport of this to 3.11, which
shouldn't be very hard. If you (or anyone else) feel adventurous and is willing
to try a backport [~rushman] I'm happy to review it.
> OutOfMemoryError when bootstrapping with less than 100GB RAM
> ------------------------------------------------------------
>
> Key: CASSANDRA-14239
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14239
> Project: Cassandra
> Issue Type: Bug
> Environment: Details of the bootstrapping Node
> * ProLiant BL460c G7
> * 56GB RAM
> * 2x 146GB 10K HDD (One dedicated for Commitlog, one for Data, Hints and
> saved_caches)
> * CentOS 7.4 on SD-Card
> * /tmp and /var/log on tmpfs
> * Oracle JDK 1.8.0_151
> * Cassandra 3.11.1
> Cluster
> * 10 existing Nodes (Up and Normal)
> Reporter: Jürgen Albersdorfer
> Priority: Major
> Attachments: Objects-by-class.csv,
> Objects-with-biggest-retained-size.csv, cassandra-env.sh, cassandra.yaml,
> gc.log.0.201804111524.zip, gc.log.0.current.zip, gc.log.201804111141.zip,
> jvm.options, jvm_opts.txt, stack-traces.txt
>
>
> Hi, I face an issue when bootstrapping a Node having less than 100GB RAM on
> our 10 Node C* 3.11.1 Cluster.
> During bootstrap, when I watch the cassandra.log I observe a growth in JVM
> Heap Old Gen which gets not significantly freed up any more.
> I know that JVM collects on Old Gen only when really needed. I can see
> collections, but there is always a remainder which seems to grow forever
> without ever getting freed.
> After the Node successfully Joined the Cluster, I can remove the extra RAM I
> have given it for bootstrapping without any further effect.
> It feels like Cassandra will not forget about every single byte streamed over
> the Network over time during bootstrapping, - which would be a memory leak
> and a major problem, too.
> I was able to produce a HeapDumpOnOutOfMemoryError from a 56GB Node (40 GB
> assigned JVM Heap). YourKit Profiler shows huge amount of Memory allocated
> for org.apache.cassandra.db.Memtable (22 GB)
> org.apache.cassandra.db.rows.BufferCell (19 GB) and java.nio.HeapByteBuffer
> (11 GB)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
