[
https://issues.apache.org/jira/browse/CASSANDRA-12413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417012#comment-15417012
]
Marcus Eriksson edited comment on CASSANDRA-12413 at 8/11/16 10:49 AM:
-----------------------------------------------------------------------
have been multiplexing this with debug logs (3x250 times) - it failed once with
this:
{code}
ERROR [Strong-Reference-Leak-Detector:1] 2016-08-10 16:32:51,086 Strong
self-ref loop detected
[/home/automaton/cassandra/build/test/cassandra/data:0/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/mc-10-big,
private volatile java.lang.Runnable
org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.obsoletion-org.apache.cassandra.db.lifecycle.LogTransaction$SSTableTidier,
private final org.apache.cassandra.db.lifecycle.Tracker
org.apache.cassandra.db.lifecycle.LogTransaction$SSTableTidier.tracker-org.apache.cassandra.db.lifecycle.Tracker,
public final java.util.Collection
org.apache.cassandra.db.lifecycle.Tracker.subscribers-java.util.concurrent.CopyOnWriteArrayList,
public final java.util.Collection
org.apache.cassandra.db.lifecycle.Tracker.subscribers-org.apache.cassandra.db.compaction.CompactionStrategyManager,
public final org.apache.cassandra.db.compaction.CompactionLogger
org.apache.cassandra.db.compaction.CompactionStrategyManager.compactionLogger-org.apache.cassandra.db.compaction.CompactionLogger,
private final org.apache.cassandra.db.ColumnFamilyStore
org.apache.cassandra.db.compaction.CompactionLogger.cfs-org.apache.cassandra.db.ColumnFamilyStore,
public final org.apache.cassandra.db.Keyspace
org.apache.cassandra.db.ColumnFamilyStore.keyspace-org.apache.cassandra.db.Keyspace,
private final java.util.concurrent.ConcurrentMap
org.apache.cassandra.db.Keyspace.columnFamilyStores-java.util.concurrent.ConcurrentHashMap,
private final java.util.concurrent.ConcurrentMap
org.apache.cassandra.db.Keyspace.columnFamilyStores-org.apache.cassandra.db.ColumnFamilyStore,
private final org.apache.cassandra.db.lifecycle.Tracker
org.apache.cassandra.db.ColumnFamilyStore.data-org.apache.cassandra.db.lifecycle.Tracker,
final java.util.concurrent.atomic.AtomicReference
org.apache.cassandra.db.lifecycle.Tracker.view-java.util.concurrent.atomic.AtomicReference,
private volatile java.lang.Object
java.util.concurrent.atomic.AtomicReference.value-org.apache.cassandra.db.lifecycle.View,
public final java.util.List
org.apache.cassandra.db.lifecycle.View.liveMemtables-com.google.common.collect.SingletonImmutableList,
final transient java.lang.Object
com.google.common.collect.SingletonImmutableList.element-org.apache.cassandra.db.Memtable,
private final org.apache.cassandra.utils.memory.MemtableAllocator
org.apache.cassandra.db.Memtable.allocator-org.apache.cassandra.utils.memory.NativeAllocator,
private final org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator
org.apache.cassandra.utils.memory.MemtableAllocator.onHeap-org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator,
private final org.apache.cassandra.utils.memory.MemtablePool$SubPool
org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.parent-org.apache.cassandra.utils.memory.MemtablePool$SubPool,
final org.apache.cassandra.utils.memory.MemtablePool
org.apache.cassandra.utils.memory.MemtablePool$SubPool.this$0-org.apache.cassandra.utils.memory.NativePool,
final org.apache.cassandra.utils.memory.MemtableCleanerThread
org.apache.cassandra.utils.memory.MemtablePool.cleaner-org.apache.cassandra.utils.memory.MemtableCleanerThread,
private java.lang.ThreadGroup java.lang.Thread.group-java.lang.ThreadGroup,
java.lang.Thread[] java.lang.ThreadGroup.threads-[Ljava.lang.Thread;,
java.lang.Thread[]
java.lang.ThreadGroup.threads-io.netty.util.concurrent.FastThreadLocalThread,
private java.lang.Runnable
java.lang.Thread.target-java.util.concurrent.ThreadPoolExecutor$Worker, final
java.util.concurrent.ThreadPoolExecutor
java.util.concurrent.ThreadPoolExecutor$Worker.this$0-org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor,
private final java.util.concurrent.BlockingQueue
java.util.concurrent.ThreadPoolExecutor.workQueue-java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue,
private final java.util.concurrent.BlockingQueue
java.util.concurrent.ThreadPoolExecutor.workQueue-java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask,
private java.util.concurrent.Callable
java.util.concurrent.FutureTask.callable-java.util.concurrent.Executors$RunnableAdapter,
final java.lang.Runnable
java.util.concurrent.Executors$RunnableAdapter.task-org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1,
final org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.this$0-org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier,
private org.apache.cassandra.utils.concurrent.Ref
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.globalRef-org.apache.cassandra.utils.concurrent.Ref]
{code}
So I made weak references to the column family store and compaction strategy
manager [here|https://github.com/krummas/cassandra/commits/marcuse/12413] to
avoid the strong reference loops (on top of the debug logging patch, only the
last commit is related to this ticket)
tests:
http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-12413-dtest/
http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-12413-testall/
also ran it 250 times
[here|http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-debuglogtest-3.9-testall/4/]
was (Author: krummas):
have been multiplexing this with debug logs (3x250 times) - it failed once with
this:
{code}
ERROR [Strong-Reference-Leak-Detector:1] 2016-08-10 16:32:51,086 Strong
self-ref loop detected
[/home/automaton/cassandra/build/test/cassandra/data:0/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/mc-10-big,
private volatile java.lang.Runnable
org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.obsoletion-org.apache.cassandra.db.lifecycle.LogTransaction$SSTableTidier,
private final org.apache.cassandra.db.lifecycle.Tracker
org.apache.cassandra.db.lifecycle.LogTransaction$SSTableTidier.tracker-org.apache.cassandra.db.lifecycle.Tracker,
public final java.util.Collection
org.apache.cassandra.db.lifecycle.Tracker.subscribers-java.util.concurrent.CopyOnWriteArrayList,
public final java.util.Collection
org.apache.cassandra.db.lifecycle.Tracker.subscribers-org.apache.cassandra.db.compaction.CompactionStrategyManager,
public final org.apache.cassandra.db.compaction.CompactionLogger
org.apache.cassandra.db.compaction.CompactionStrategyManager.compactionLogger-org.apache.cassandra.db.compaction.CompactionLogger,
private final org.apache.cassandra.db.ColumnFamilyStore
org.apache.cassandra.db.compaction.CompactionLogger.cfs-org.apache.cassandra.db.ColumnFamilyStore,
public final org.apache.cassandra.db.Keyspace
org.apache.cassandra.db.ColumnFamilyStore.keyspace-org.apache.cassandra.db.Keyspace,
private final java.util.concurrent.ConcurrentMap
org.apache.cassandra.db.Keyspace.columnFamilyStores-java.util.concurrent.ConcurrentHashMap,
private final java.util.concurrent.ConcurrentMap
org.apache.cassandra.db.Keyspace.columnFamilyStores-org.apache.cassandra.db.ColumnFamilyStore,
private final org.apache.cassandra.db.lifecycle.Tracker
org.apache.cassandra.db.ColumnFamilyStore.data-org.apache.cassandra.db.lifecycle.Tracker,
final java.util.concurrent.atomic.AtomicReference
org.apache.cassandra.db.lifecycle.Tracker.view-java.util.concurrent.atomic.AtomicReference,
private volatile java.lang.Object
java.util.concurrent.atomic.AtomicReference.value-org.apache.cassandra.db.lifecycle.View,
public final java.util.List
org.apache.cassandra.db.lifecycle.View.liveMemtables-com.google.common.collect.SingletonImmutableList,
final transient java.lang.Object
com.google.common.collect.SingletonImmutableList.element-org.apache.cassandra.db.Memtable,
private final org.apache.cassandra.utils.memory.MemtableAllocator
org.apache.cassandra.db.Memtable.allocator-org.apache.cassandra.utils.memory.NativeAllocator,
private final org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator
org.apache.cassandra.utils.memory.MemtableAllocator.onHeap-org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator,
private final org.apache.cassandra.utils.memory.MemtablePool$SubPool
org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.parent-org.apache.cassandra.utils.memory.MemtablePool$SubPool,
final org.apache.cassandra.utils.memory.MemtablePool
org.apache.cassandra.utils.memory.MemtablePool$SubPool.this$0-org.apache.cassandra.utils.memory.NativePool,
final org.apache.cassandra.utils.memory.MemtableCleanerThread
org.apache.cassandra.utils.memory.MemtablePool.cleaner-org.apache.cassandra.utils.memory.MemtableCleanerThread,
private java.lang.ThreadGroup java.lang.Thread.group-java.lang.ThreadGroup,
java.lang.Thread[] java.lang.ThreadGroup.threads-[Ljava.lang.Thread;,
java.lang.Thread[]
java.lang.ThreadGroup.threads-io.netty.util.concurrent.FastThreadLocalThread,
private java.lang.Runnable
java.lang.Thread.target-java.util.concurrent.ThreadPoolExecutor$Worker, final
java.util.concurrent.ThreadPoolExecutor
java.util.concurrent.ThreadPoolExecutor$Worker.this$0-org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor,
private final java.util.concurrent.BlockingQueue
java.util.concurrent.ThreadPoolExecutor.workQueue-java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue,
private final java.util.concurrent.BlockingQueue
java.util.concurrent.ThreadPoolExecutor.workQueue-java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask,
private java.util.concurrent.Callable
java.util.concurrent.FutureTask.callable-java.util.concurrent.Executors$RunnableAdapter,
final java.lang.Runnable
java.util.concurrent.Executors$RunnableAdapter.task-org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1,
final org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.this$0-org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier,
private org.apache.cassandra.utils.concurrent.Ref
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.globalRef-org.apache.cassandra.utils.concurrent.Ref]
{code}
So I made weak references to the column family store and compaction strategy
manager [here|https://github.com/krummas/cassandra/commits/marcuse/12413] to
avoid the strong reference loops
tests:
http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-12413-dtest/
http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-12413-testall/
also ran it 250 times
[here|http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-debuglogtest-3.9-testall/4/]
> CompactionsCQLTest.testTriggerMinorCompactionDTCS fails
> -------------------------------------------------------
>
> Key: CASSANDRA-12413
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12413
> Project: Cassandra
> Issue Type: Bug
> Reporter: Joshua McKenzie
> Assignee: Marcus Eriksson
> Labels: unittest
> Fix For: 3.9
>
>
> [Link|http://cassci.datastax.com/job/cassandra-3.9_testall/lastCompletedBuild/testReport/org.apache.cassandra.db.compaction/CompactionsCQLTest/testTriggerMinorCompactionDTCS/]
> Error Message
> No minor compaction triggered in 5000ms
> Stacktrace
> {noformat}
> junit.framework.AssertionFailedError: No minor compaction triggered in 5000ms
> at
> org.apache.cassandra.db.compaction.CompactionsCQLTest.waitForMinor(CompactionsCQLTest.java:247)
> at
> org.apache.cassandra.db.compaction.CompactionsCQLTest.testTriggerMinorCompactionDTCS(CompactionsCQLTest.java:72)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
