[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202502#comment-16202502
]
Dan Kinder commented on CASSANDRA-13943:
----------------------------------------
Sure, I can try, but I'm not sure how that would help, these nodes have plenty
of cpu and iops available... But there is nonetheless a problem, because
flushers are getting stuck.
Looking at the stack trace, it seems to come down to the readLock in
CompactionStrategyManager. The flushers are blocked on that:
{noformat}
Thread 7385: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int)
@bci=83, line=967 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int)
@bci=10, line=1283 (Compiled frame)
- java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5,
line=727 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionStrategyManager.createSSTableMultiWriter(org.apache.cassandra.io.sstable.Descriptor,
long, long, org.apache. cassandra.io.sstable.metadata.MetadataCollector,
org.apache.cassandra.db.SerializationHeader, java.util.Collection,
org.apache.cassandra.db.lifecycle. LifecycleTransaction) @bci=4, line=901
(Compiled frame)
-
org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(org.apache.cassandra.io.sstable.Descriptor,
long, long, org.apache.cassandra.io. sstable.metadata.MetadataCollector,
org.apache.cassandra.db.SerializationHeader,
org.apache.cassandra.db.lifecycle.LifecycleTransaction) @bci=21, line=515
(Compiled frame)
-
org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(org.apache.cassandra.db.lifecycle.LifecycleTransaction,
java.lang.String, org.apache. cassandra.db.PartitionColumns,
org.apache.cassandra.db.rows.EncodingStats) @bci=104, line=506 (Compiled frame)
-
org.apache.cassandra.db.Memtable$FlushRunnable.<init>(org.apache.cassandra.db.Memtable,
java.util.concurrent.ConcurrentNavigableMap,
org.apache.cassandra.db.Directories$DataDirectory,
org.apache.cassandra.db.PartitionPosition,
org.apache.cassandra.db.PartitionPosition, org.apache.cassandra.db.lifecycle.
LifecycleTransaction) @bci=253, line=447 (Compiled frame)
-
org.apache.cassandra.db.Memtable$FlushRunnable.<init>(org.apache.cassandra.db.Memtable,
org.apache.cassandra.db.PartitionPosition, org.apache.cassandra.
db.PartitionPosition, org.apache.cassandra.db.Directories$DataDirectory,
org.apache.cassandra.db.lifecycle.LifecycleTransaction) @bci=19, line=417
(Compiled frame)
-
org.apache.cassandra.db.Memtable.createFlushRunnables(org.apache.cassandra.db.lifecycle.LifecycleTransaction)
@bci=125, line=318 (Interpreted frame)
-
org.apache.cassandra.db.Memtable.flushRunnables(org.apache.cassandra.db.lifecycle.LifecycleTransaction)
@bci=2, line=300 (Compiled frame)
-
org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(org.apache.cassandra.db.Memtable,
boolean) @bci=82, line=1137 (Compiled frame)
- org.apache.cassandra.db.ColumnFamilyStore$Flush.run() @bci=85, line=1102
(Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1149 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624
(Compiled frame)
-
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(java.lang.Runnable)
@bci=1, line=81 (Compiled frame)
- org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5.run() @bci=4
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
{noformat}
All the callers in CompactionStrategyManager that *aren't* blocked on a lock
are trying to get background tasks:
{noformat}
Thread 1468: (state = IN_JAVA)
- java.util.HashMap$HashIterator.nextNode() @bci=95, line=1441 (Compiled
frame; information may be imprecise)
- java.util.HashMap$KeyIterator.next() @bci=1, line=1461 (Compiled frame)
- java.util.AbstractCollection.toArray() @bci=39, line=141 (Compiled frame)
- java.util.ArrayList.<init>(java.util.Collection) @bci=6, line=177 (Compiled
frame)
-
org.apache.cassandra.db.compaction.LeveledManifest.ageSortedSSTables(java.util.Collection)
@bci=5, line=731 (Compiled frame)
- org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(int)
@bci=187, line=644 (Compiled frame)
- org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates()
@bci=290, line=385 (Compiled frame)
-
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(int)
@bci=4, line=119 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(int)
@bci=105, line=124 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run()
@bci=84, line=262 (Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
(Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1149 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624
(Compiled frame)
-
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(java.lang.Runnable)
@bci=1, line=81 (Compiled frame)
- org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5.run() @bci=4
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
...
Thread 1709: (state = BLOCKED)
- org.apache.cassandra.db.compaction.LeveledManifest.getLevel(int) @bci=5,
line=769 (Compiled frame)
-
org.apache.cassandra.db.compaction.LeveledManifest.getSTCSInL0CompactionCandidate()
@bci=8, line=398 (Compiled frame)
- org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates()
@bci=152, line=361 (Compiled frame)
-
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(int)
@bci=4, line=119 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(int)
@bci=105, line=124 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run()
@bci=84, line=262 (Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
(Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1149 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624
(Compiled frame)
-
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(java.lang.Runnable)
@bci=1, line=81 (Compiled frame)
- org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5.run() @bci=4
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
...
Thread 30170: (state = IN_JAVA)
-
org.apache.cassandra.db.compaction.LeveledManifest.getOverlappingStarvedSSTables(int,
java.util.Collection) @bci=380, line=478 (Compiled frame;
information may be imprecise)
- org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates()
@bci=195, line=370 (Compiled frame)
-
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(int)
@bci=4, line=119 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(int)
@bci=105, line=124 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run()
@bci=84, line=262 (Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
(Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1149 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624
(Compiled frame)
-
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(java.lang.Runnable)
@bci=1, line=81 (Compiled frame)
- org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5.run() @bci=4
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
...
Thread 31837: (state = IN_VM)
- org.apache.cassandra.db.compaction.LeveledManifest.getLevel(int) @bci=5,
line=769 (Compiled frame; information may be imprecise)
-
org.apache.cassandra.db.compaction.LeveledManifest.getSTCSInL0CompactionCandidate()
@bci=8, line=398 (Compiled frame)
- org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates()
@bci=152, line=361 (Compiled frame)
-
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(int)
@bci=4, line=119 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(int)
@bci=105, line=124 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run()
@bci=84, line=262 (Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
(Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1149 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624
(Compiled frame)
-
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(java.lang.Runnable)
@bci=1, line=81 (Compiled frame)
- org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5.run() @bci=4
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
... etc.
{noformat}
Since these threads are stopped at different points, it leads me to assume that
they are not deadlocked at one spot, but that
BackgroundCompactionCandidate.run() may be getting run far too frequently or
infinitely.
This matches the symptoms in CASSANDRA-13923 -- maybe that one is actually not
a duplicate of CASSANDRA-13215 ?
> Infinite compaction of L0 SSTables in JBOD
> ------------------------------------------
>
> Key: CASSANDRA-13943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.11.0 / Centos 6
> Reporter: Dan Kinder
> Assignee: Marcus Eriksson
> Attachments: cassandra-jstack-2017-10-12-infinite-sstable-adding.txt,
> cassandra-jstack-2017-10-12.txt, cassandra.yaml, debug.log,
> debug.log-with-commit-d8f3f2780, debug.log.1.zip, debug.log.zip, jvm.options
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over.
> Attaching logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It
> does create new SSTables but immediately recompacts again. Note that I am not
> inserting anything at the moment, there is no flushing happening on this
> table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates.
> But they shouldn't be, they are on different disks and I ran nodetool
> relocatesstables as well as nodetool compact. So, it tries to compact them
> together, but the compaction results in the exact same 2 SSTables on the 2
> disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU
> doing this for multiple tables, it seems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
