[
https://issues.apache.org/jira/browse/CASSANDRA-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047582#comment-15047582
]
Paulo Motta commented on CASSANDRA-10797:
-----------------------------------------
I was able to reproduce the bootstrapping OOM locally on CCM with 1000 sstables
and a 50MB heap. We might be able to reduce significantly the stream receiving
task footprint by closing the {{SStableWriters}} as soon as they're finished
and keeping only the temporary descriptors, finalizing/renaming them after the
stream is completed (and removing them if the task is aborted).
I'm not sure if this will really solve the problem though, as in the end of the
stream task the finalized {{SStableReaders}} will be loaded and added to the
{{DataTracker}}, so the node will need to have sufficient memory to hold those
readers, what might still be a problem if there are tens of thousands of them.
I will test this approach locally and report back the results.
> Bootstrap new node fails with OOM when streaming nodes contains thousands of
> sstables
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-10797
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10797
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Cassandra 2.1.8.621 w/G1GC
> Reporter: Jose Martinez Poblete
> Assignee: Paulo Motta
> Fix For: 2.1.x
>
> Attachments: 112415_system.log, Heapdump_OOM.zip, Screen Shot
> 2015-12-01 at 7.34.40 PM.png
>
>
> When adding a new node to an existing DC, it runs OOM after 25-45 minutes
> Upon heapdump revision, it is found the sending nodes are streaming thousands
> of sstables which in turns blows the bootstrapping node heap
> {noformat}
> ERROR [RMI Scheduler(0)] 2015-11-24 10:10:44,585
> JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting
> forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> ERROR [STREAM-IN-/173.36.28.148] 2015-11-24 10:10:44,585
> StreamSession.java:502 - [Stream #0bb13f50-92cb-11e5-bc8d-f53b7528ffb4]
> Streaming error occurred
> java.lang.IllegalStateException: Shutdown in progress
> at
> java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
> ~[na:1.8.0_65]
> at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:747)
> ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
> at
> org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:95)
> ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
> at
> org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:64)
> ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:66)
> ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38)
> ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
> ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> ERROR [RMI TCP Connection(idle)] 2015-11-24 10:10:44,585
> JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting
> forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> ERROR [OptionalTasks:1] 2015-11-24 10:10:44,585 CassandraDaemon.java:223 -
> Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.IllegalStateException: Shutdown in progress
> {noformat}
> Attached is the Eclipse MAT report as a zipped web page
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
