[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862512#comment-15862512
]
Jay Zhuang commented on CASSANDRA-13008:
Thanks [~Stefania] for the review. You're right, no need to check if it's not
{{mmap}}:
https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/util/CompressedSegmentedFile.java#L46
> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in
> cluster crashed. As each node was trying to load more local SSTable files for
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to
> be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861194#comment-15861194
]
Stefania commented on CASSANDRA-13008:
--
On second thought, I don't think we need this warning if the disk access mode
is not mmap, can you
[check|https://github.com/stef1927/cassandra/commit/7d4139fb50e8ef11b824138dd72bde036b9ee152#diff-a5df240149285ae528cdd3c41aa59360R240]
[~jay.zhuang]?
> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in
> cluster crashed. As each node was trying to load more local SSTable files for
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to
> be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}
--
This message was sent by Atlassian JIRA
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859854#comment-15859854
]
Stefania commented on CASSANDRA-13008:
--
Thanks for the patch and sorry for the delay with the review.
The patch LGTM - I've merely fixed the brackets positions and replaced
{{FBUtilities.isWindows}} with {{FBUtilities.hasProcFS()}}.
I've started running the tests on 3.0 only:
||3.0||3.11||trunk||
|[patch|https://github.com/stef1927/cassandra/tree/13008-3.0]|[patch|https://github.com/stef1927/cassandra/tree/13008-3.11]|[patch|https://github.com/stef1927/cassandra/tree/13008]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-3.11-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-3.11-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-dtest/]|
The 3.0 patch applies cleanly to 3.11 and trunk.
> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in
> cluster crashed. As each node was trying to load more local SSTable files for
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to
> be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
>
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854684#comment-15854684
]
Jay Zhuang commented on CASSANDRA-13008:
Thanks.
> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in
> cluster crashed. As each node was trying to load more local SSTable files for
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to
> be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853687#comment-15853687
]
Stefania commented on CASSANDRA-13008:
--
Sure, I'll try to review it in the next few days.
> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in
> cluster crashed. As each node was trying to load more local SSTable files for
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to
> be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853296#comment-15853296
]
Jay Zhuang commented on CASSANDRA-13008:
[~Stefania] would you please review this patch?
> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in
> cluster crashed. As each node was trying to load more local SSTable files for
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to
> be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761849#comment-15761849
]
Jay Zhuang commented on CASSANDRA-13008:
Could someone please review this change?
> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in
> cluster crashed. As each node was trying to load more local SSTable files for
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to
> be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
