Could the problem be that the process ran out of file handles? Recommendation is to tune that higher than the default.
Hannu > onmstester onmstester <onmstes...@zoho.com> kirjoitti 12.7.2018 kello 12.44: > > Cassandra crashed in Two out of 10 nodes in my cluster within 1 day, the > error is: > > ERROR [CompactionExecutor:3389] 2018-07-10 11:27:58,857 > CassandraDaemon.java:228 - Exception in thread > Thread[CompactionExecutor:3389,1,main] > org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed > at > org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:275) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_65] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_65] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_65] > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) > [apache-cassandra-3.11.2.jar:3.11.2] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] > Caused by: java.io.IOException: Map failed > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) > ~[na:1.8.0_65] > at > org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) > ~[apache-cassandra-3.11.2.jar:3.11.2] > ... 23 common frames omitted > Caused by: java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_65] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) > ~[na:1.8.0_65] > ... 24 common frames omitted > > Each node has 128 GB ram which 32 GB allocated as Cassandra Heap. > Sent using Zoho Mail > > >