[
https://issues.apache.org/jira/browse/CASSANDRA-16243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-16243:
------------------------------------
Bug Category: Parent values: Degradation(12984)Level 1 values: Resource
Management(12995)
Complexity: Challenging
Component/s: Local/Compaction
Discovered By: User Report
Severity: Critical
Status: Open (was: Triage Needed)
> LEAK DETECTED: was not released before the reference was garbage collected
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-16243
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16243
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Reporter: developthou
> Priority: Normal
>
> Hello,
> I am getting the following errors and I am running 3.11.5
> {code:java}
> LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@356cb07f) to class
> org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1709177197:Memory@[7e7a3b64b910..7e7a3b96b910)
> was not released before the reference was garbage collected
> ERROR [CompactionExecutor:3763] 2020-11-02 02:34:53,316
> CassandraDaemon.java:228 - Exception in thread
> Thread[CompactionExecutor:3763,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
> [apache-cassandra-3.11.5.jar:3.11.5]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936)
> ~[na:1.8.0_60]
> ... 24 common frames omitted
> WARN [GossipTasks:1] 2020-11-02 02:34:53,302 FailureDetector.java:278 - Not
> marking nodes down due to local pause of 7468254535 > 5000000000
> ERROR [Reference-Reaper] 2020-11-02 02:34:53,208 Ref.java:229 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@33fef449) to class
> org.apache.cassandra.io.util.FileHandle$Cleanup@1098210945:/data/cassandra/data/elvisevallogks/decision_detail-7351f6205f3811ea922ce1efeeba3e49/md-3695-big-Index.db
> was not released before the reference was garbage collected
> ERROR [CompactionExecutor:3762] 2020-11-02 02:34:53,208
> CassandraDaemon.java:228 - Exception in thread
> Thread[CompactionExecutor:3762,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
> [apache-cassandra-3.11.5.jar:3.11.5]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936)
> ~[na:1.8.0_60]
> ... 24 common frames omitted
> INFO [Service Thread] 2020-11-02 02:34:53,877 StatusLogger.java:51 -
> MemtablePostFlush 0 0 3596 0
> 0 {code}
>
> These are the settings
> {code:java}
> # /etc/security/limits-cassandra.conf
> # Cassandra-specfic parameters
> # add
> cassandra soft memlock unlimited
> cassandra hard memlock unlimited
> cassandra soft nproc 65535
> #change
> * soft nofile 655350
> * hard nofile 655350 {code}
> For cassandra process
> {code:java}
> cat /proc/25586/limits
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size 8388608 unlimited bytes
> Max core file size 0 0 bytes
> Max resident set unlimited unlimited bytes
> Max processes 65535 692334 processes
> Max open files 655350 655350 files
> Max locked memory unlimited unlimited bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 692334 692334 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us {code}
> {{vm.max_map_count is set to 1048575}}
> {code:java}
> sysctl vm.max_map_count
> vm.max_map_count = 1048575{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
