[
https://issues.apache.org/jira/browse/CASSANDRA-16242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabin Shrestha updated CASSANDRA-16242:
----------------------------------------
Description:
Hello,
I am getting the following errors and I am running 3.11.5
{code:java}
LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@356cb07f) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1709177197:Memory@[7e7a3b64b910..7e7a3b96b910)
was not released before the reference was garbage collected
ERROR [CompactionExecutor:3763] 2020-11-02 02:34:53,316
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:3763,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
[apache-cassandra-3.11.5.jar:3.11.5]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939)
~[na:1.8.0_60]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.5.jar:3.11.5]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936)
~[na:1.8.0_60]
... 24 common frames omitted
WARN [GossipTasks:1] 2020-11-02 02:34:53,302 FailureDetector.java:278 - Not
marking nodes down due to local pause of 7468254535 > 5000000000
ERROR [Reference-Reaper] 2020-11-02 02:34:53,208 Ref.java:229 - LEAK DETECTED:
a reference (org.apache.cassandra.utils.concurrent.Ref$State@33fef449) to class
org.apache.cassandra.io.util.FileHandle$Cleanup@1098210945:/data/cassandra/data/elvisevallogks/decision_detail-7351f6205f3811ea922ce1efeeba3e49/md-3695-big-Index.db
was not released before the reference was garbage collected
ERROR [CompactionExecutor:3762] 2020-11-02 02:34:53,208
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:3762,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
[apache-cassandra-3.11.5.jar:3.11.5]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939)
~[na:1.8.0_60]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.5.jar:3.11.5]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936)
~[na:1.8.0_60]
... 24 common frames omitted
INFO [Service Thread] 2020-11-02 02:34:53,877 StatusLogger.java:51 -
MemtablePostFlush 0 0 3596 0
0 {code}
These are the settings
{code:java}
# /etc/security/limits-cassandra.conf
# Cassandra-specfic parameters
# add
cassandra soft memlock unlimited
cassandra hard memlock unlimited
cassandra soft nproc 65535
#change
* soft nofile 655350
* hard nofile 655350 {code}
For cassandra process
{code:java}
cat /proc/25586/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 65535 692334 processes
Max open files 655350 655350 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 692334 692334 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us {code}
{{vm.max_map_count is set to 1048575}}
{code:java}
sysctl vm.max_map_count
vm.max_map_count = 1048575 {code}
was:
Hello,
I am getting the following errors and I am running 3.11.5
{code:java}
LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@356cb07f) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1709177197:Memory@[7e7a3b64b910..7e7a3b96b910)
was not released before the reference was garbage collected
ERROR [CompactionExecutor:3763] 2020-11-02 02:34:53,316
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:3763,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
[apache-cassandra-3.11.5.jar:3.11.5]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939)
~[na:1.8.0_60]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.5.jar:3.11.5]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936)
~[na:1.8.0_60]
... 24 common frames omitted
WARN [GossipTasks:1] 2020-11-02 02:34:53,302 FailureDetector.java:278 - Not
marking nodes down due to local pause of 7468254535 > 5000000000
ERROR [Reference-Reaper] 2020-11-02 02:34:53,208 Ref.java:229 - LEAK DETECTED:
a reference (org.apache.cassandra.utils.concurrent.Ref$State@33fef449) to class
org.apache.cassandra.io.util.FileHandle$Cleanup@1098210945:/data/cassandra/data/elvisevallogks/decision_detail-7351f6205f3811ea922ce1efeeba3e49/md-3695-big-Index.db
was not released before the reference was garbage collected
ERROR [CompactionExecutor:3762] 2020-11-02 02:34:53,208
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:3762,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
[apache-cassandra-3.11.5.jar:3.11.5]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939)
~[na:1.8.0_60]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.5.jar:3.11.5]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936)
~[na:1.8.0_60]
... 24 common frames omitted
INFO [Service Thread] 2020-11-02 02:34:53,877 StatusLogger.java:51 -
MemtablePostFlush 0 0 3596 0
0 {code}
These are the settings
# /etc/security/limits-cassandra.conf
# Cassandra-specfic parameters
# add
cassandra soft memlock unlimited
cassandra hard memlock unlimited
cassandra soft nproc 65535
#change
* soft nofile 655350
* hard nofile 655350
For cassandra process
cat /proc/25586/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 65535 692334 processes
Max open files 655350 655350 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 692334 692334 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
{{vm.max_map_count is set to 1048575}}
{code:java}
sysctl vm.max_map_count
vm.max_map_count = 1048575 {code}
> LEAK DETECTED: was not released before the reference was garbage collected
> ---------------------------------------------------------------------------
>
> Key: CASSANDRA-16242
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16242
> Project: Cassandra
> Issue Type: Task
> Reporter: Prabin Shrestha
> Priority: Normal
>
> Hello,
> I am getting the following errors and I am running 3.11.5
> {code:java}
> LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@356cb07f) to class
> org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1709177197:Memory@[7e7a3b64b910..7e7a3b96b910)
> was not released before the reference was garbage collected
> ERROR [CompactionExecutor:3763] 2020-11-02 02:34:53,316
> CassandraDaemon.java:228 - Exception in thread
> Thread[CompactionExecutor:3763,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
> [apache-cassandra-3.11.5.jar:3.11.5]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936)
> ~[na:1.8.0_60]
> ... 24 common frames omitted
> WARN [GossipTasks:1] 2020-11-02 02:34:53,302 FailureDetector.java:278 - Not
> marking nodes down due to local pause of 7468254535 > 5000000000
> ERROR [Reference-Reaper] 2020-11-02 02:34:53,208 Ref.java:229 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@33fef449) to class
> org.apache.cassandra.io.util.FileHandle$Cleanup@1098210945:/data/cassandra/data/elvisevallogks/decision_detail-7351f6205f3811ea922ce1efeeba3e49/md-3695-big-Index.db
> was not released before the reference was garbage collected
> ERROR [CompactionExecutor:3762] 2020-11-02 02:34:53,208
> CassandraDaemon.java:228 - Exception in thread
> Thread[CompactionExecutor:3762,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
> [apache-cassandra-3.11.5.jar:3.11.5]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936)
> ~[na:1.8.0_60]
> ... 24 common frames omitted
> INFO [Service Thread] 2020-11-02 02:34:53,877 StatusLogger.java:51 -
> MemtablePostFlush 0 0 3596 0
> 0 {code}
>
> These are the settings
> {code:java}
> # /etc/security/limits-cassandra.conf
> # Cassandra-specfic parameters
> # add
> cassandra soft memlock unlimited
> cassandra hard memlock unlimited
> cassandra soft nproc 65535
> #change
> * soft nofile 655350
> * hard nofile 655350 {code}
> For cassandra process
> {code:java}
> cat /proc/25586/limits
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size 8388608 unlimited bytes
> Max core file size 0 0 bytes
> Max resident set unlimited unlimited bytes
> Max processes 65535 692334 processes
> Max open files 655350 655350 files
> Max locked memory unlimited unlimited bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 692334 692334 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us {code}
> {{vm.max_map_count is set to 1048575}}
> {code:java}
> sysctl vm.max_map_count
> vm.max_map_count = 1048575 {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
