Probably close - maybe file handles or map counts. ulimit -a and/or
cat /proc/sys/vm/max_map_count
Would be useful
--
Jeff Jirsa
> On Jul 12, 2018, at 3:47 AM, Hannu Kröger wrote:
>
> Could the problem be that the process ran out of file handles? Recommendation
> is to tune that higher than the default.
>
> Hannu
>
>> onmstester onmstester kirjoitti 12.7.2018 kello 12.44:
>>
>> Cassandra crashed in Two out of 10 nodes in my cluster within 1 day, the
>> error is:
>>
>> ERROR [CompactionExecutor:3389] 2018-07-10 11:27:58,857
>> CassandraDaemon.java:228 - Exception in thread
>> Thread[CompactionExecutor:3389,1,main]
>> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
>> at
>> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:275)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[na:1.8.0_65]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[na:1.8.0_65]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[na:1.8.0_65]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_65]
>> at
>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>> [apache-cassandra-3.11.2.jar:3.11.2]
>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
>> Caused by: java.io.IOException: Map failed
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939)
>> ~[na:1.8.0_65]
>> at
>> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>> ... 23 common frames omitted
>> Caused by: java.lang.OutOfMemoryError: Map failed
>> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_65]
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936)
>> ~[na:1.8.0_65]
>> ... 24 common frames omitted
>>
>> Each node has 128 GB ram which 32 GB allocated as Cassandra Heap.
>> Sent using Zoho Mail
>>
>>
>>