[
https://issues.apache.org/jira/browse/CASSANDRA-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546363#comment-15546363
]
Mikhail Krupitskiy commented on CASSANDRA-12662:
------------------------------------------------
We've done a parallel experiment on another cluster (please let me know if it
would be better to move it to a separate issue):
Cassandra 3.5 cluster with 3 nodes and replication factor = 3.
Each node: HDD, 4 CPU cores, 16g RAM, Xmx=4g.
During a period of intensive write load nodes failed with OOM described in the
issue.
As a try to solve it we decreased 'max_compaction_flush_memory_in_mb' for our
SASI indexes to 512Mb.
After that nodes started to fail with the following:
{noformat}
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.index.sasi.utils.MappedBuffer.<init>(MappedBuffer.java:78)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.index.sasi.utils.MappedBuffer.<init>(MappedBuffer.java:57)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:142)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$scheduleSegmentFlush$261(PerSSTableIndexWriter.java:262)
~[apache-cassandra-3.5.jar:3.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_91]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
~[apache-cassandra-3.5.jar:3.5]
... 8 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_91]
... 9 common frames omitted
{noformat}
> OOM when using SASI index
> -------------------------
>
> Key: CASSANDRA-12662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12662
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes
> ~8Gb, of which ~4Gb is Java heap
> Reporter: Maxim Podkolzine
> Priority: Critical
> Fix For: 3.x
>
> Attachments: memory-dump.png
>
>
> 2.8Gb of the heap is taken by the index data, pending for flush (see the
> screenshot). As a result the node fails with OOM.
> Questions:
> - Why can't Cassandra keep up with the inserted data and flush it?
> - What resources/configuration should be changed to improve the performance?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
