Probably close - maybe file handles or map counts. ulimit -a and/or cat /proc/sys/vm/max_map_count
Would be useful -- Jeff Jirsa > On Jul 12, 2018, at 3:47 AM, Hannu Kröger <hkro...@gmail.com> wrote: > > Could the problem be that the process ran out of file handles? Recommendation > is to tune that higher than the default. > > Hannu > >> onmstester onmstester <onmstes...@zoho.com> kirjoitti 12.7.2018 kello 12.44: >> >> Cassandra crashed in Two out of 10 nodes in my cluster within 1 day, the >> error is: >> >> ERROR [CompactionExecutor:3389] 2018-07-10 11:27:58,857 >> CassandraDaemon.java:228 - Exception in thread >> Thread[CompactionExecutor:3389,1,main] >> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed >> at >> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:275) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> ~[na:1.8.0_65] >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> ~[na:1.8.0_65] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> ~[na:1.8.0_65] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_65] >> at >> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) >> [apache-cassandra-3.11.2.jar:3.11.2] >> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] >> Caused by: java.io.IOException: Map failed >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) >> ~[na:1.8.0_65] >> at >> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> ... 23 common frames omitted >> Caused by: java.lang.OutOfMemoryError: Map failed >> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_65] >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) >> ~[na:1.8.0_65] >> ... 24 common frames omitted >> >> Each node has 128 GB ram which 32 GB allocated as Cassandra Heap. >> Sent using Zoho Mail >> >> >>
