[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711207#comment-16711207
]
Marcus Eriksson commented on CASSANDRA-14908:
-
bq. Do you think the issue will be related to repair ?
Incremental repair used to cause issues like this
Could you build and deploy a version of cassandra with more logging
information, if so I'll create a patch for you?
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0
> tid=0x7f5a3d4ba000 nid=0x24063 waiting for monitor entry
> [0x7fa8337bd000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
>
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711168#comment-16711168
]
wu taiyin commented on CASSANDRA-14908:
---
Hello +{color:#0066cc}Marcus Eriksson{color}+
we dont run the repair , and i also think that the repair task will not use the
function:{color:#ff}getCompactionCandidates.{color}
{color:#33}Do you think the issue will be related to repair ? what is the
relationship between them ?{color}
{color:#33}wish your reply.{color}
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0
> tid=0x7f5a3d4ba000 nid=0x24063 waiting for monitor entry
> [0x7fa8337bd000]
>
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710214#comment-16710214
]
Marcus Eriksson commented on CASSANDRA-14908:
-
[~dockerwu] do you run repair at all? If so, how?
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0
> tid=0x7f5a3d4ba000 nid=0x24063 waiting for monitor entry
> [0x7fa8337bd000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704580#comment-16704580
]
wu taiyin commented on CASSANDRA-14908:
---
Thank you for you reply. and the environment as follows:
CPU：48C memory：385G
disk and data size per disk:
/dev/sda3 754G 9.2G 706G 2% /opt/huawei
/dev/sdb 1.5T 223G 1.3T 15% /disk/data1
/dev/sdc 1.5T 223G 1.3T 15% /disk/data2
/dev/sdd 1.5T 213G 1.3T 15% /disk/data3
/dev/sde 1.5T 216G 1.3T 15% /disk/data4
/dev/sdf 1.5T 214G 1.3T 15% /disk/data5
Cassandra.yaml :
data_file_directories:
- /disk/data1/cassandra
- /disk/data2/cassandra
- /disk/data3/cassandra
- /disk/data4/cassandra
- /disk/data5/cassandra
commitlog_directory: /opt/huawei/logs/cassandra/commitlog
./nodetool cfstats app99
Keyspace: app99
Read Count: 2789855752
Read Latency: 0.6304968936945955 ms.
Write Count: 1490782843
Write Latency: 0.03916580592616869 ms.
Pending Flushes: 0
Table: dn
SSTable count: 1
SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 6092
Space used (total): 6092
Space used by snapshots (total): 0
Off heap memory used (total): 127
SSTable Compression Ratio: 0.14210985178727115
Number of keys (estimate): 11
Memtable cell count: 483
Memtable data size: 2352
Memtable off heap memory used: 0
Memtable switch count: 2
Local read count: 10133694
Local read latency: 0.136 ms
Local write count: 270
Local write latency: 0.135 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 96
Bloom filter off heap memory used: 88
Index summary off heap memory used: 31
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 125
Compacted partition maximum bytes: 1916
Compacted partition mean bytes: 974
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: file
SSTable count: 4295
SSTables in each level: [0, 10, 100, 1016/1000, 3169, 0, 0, 0, 0]
Space used (live): 1166100476265
Space used (total): 1169018887208
Space used by snapshots (total): 0
Off heap memory used (total): 9184778389
SSTable Compression Ratio: 0.38421502411531816
Number of keys (estimate): 8849691024
Memtable cell count: 8238732
Memtable data size: 301883508
Memtable off heap memory used: 0
Memtable switch count: 521
Local read count: 2779722909
Local read latency: 0.633 ms
Local write count: 1490783015
Local write latency: 0.040 ms
Pending flushes: 0
Bloom filter false positives: 5092
Bloom filter false ratio: 0.0
Bloom filter space used: 6387737864
Bloom filter off heap memory used: 6387703504
Index summary off heap memory used: 2575905581
Compression metadata off heap memory used: 221169304
Compacted partition minimum bytes: 43
Compacted partition maximum bytes: 88148
Compacted partition mean bytes: 215
Average live cells per slice (last five minutes): 0.4247412120152055
Maximum live cells per slice (last five minutes): 1.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704492#comment-16704492
]
Marcus Eriksson commented on CASSANDRA-14908:
-
Could you describe your environment a bit more? how are you running repair, how
big are your nodes?
Issues like this have historically been caused by the leveled manifest being
wrong (having sstables in the wrong levels etc), but haven't seen any reports
like this for a long time (and 2.1.15 is more than 2 years old)
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0
> tid=0x7f5a3d4ba000 nid=0x24063 waiting for monitor entry
> [0x7fa8337bd000]
>
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704449#comment-16704449
]
wu taiyin commented on CASSANDRA-14908:
---
No exception log found . I just find that there are many pending task of
MemtableFlushWriter stage by nodetool tpstats .
and I noticed that the mutation request to this node failed to execute.
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0
> tid=0x7f5a3d4ba000 nid=0x24063 waiting for monitor entry
> [0x7fa8337bd000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
>
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704386#comment-16704386
]
Marcus Eriksson commented on CASSANDRA-14908:
-
Not aware of any bugs in 2.1 causing this right now, do you see any exceptions
in your logs?
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0
> tid=0x7f5a3d4ba000 nid=0x24063 waiting for monitor entry
> [0x7fa8337bd000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
>
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704131#comment-16704131
]
wu taiyin commented on CASSANDRA-14908:
---
We have seen it twice in the production environment, but unfortunately we have
not found the reproducible steps.
I have also seen the CASSANDRA-11373 which looks like this issue, it also loop
infinite cause by sstable reference error。
But the CASSANDRA-11373 happened in 3.*，and the one is 2.1.15.
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0
> tid=0x7f5a3d4ba000 nid=0x24063 waiting for monitor entry
> [0x7fa8337bd000]
> java.lang.Thread.State: BLOCKED (on object
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704108#comment-16704108
]
wu taiyin commented on CASSANDRA-14908:
---
Yes, i have studied the function getNextBackgroundTask. It will never return
unless it can't mark compacting candidate sstables successfully.
public Collection getMaximalTask(int gcBefore)
{
while (true)
{
//other code
{color:#FF}//it will loop infinite if markCompacting always return
false.{color}
if (cfs.getDataTracker().markCompacting(candidate.sstables))
{
LeveledCompactionTask newTask = new LeveledCompactionTask(cfs,
candidate.sstables, candidate.level, gcBefore, candidate.maxSSTableBytes);
newTask.setCompactionType(op);
return Arrays.asList(newTask);
}
}
}
And the doubt of the function "markCompacting" as follows:
public boolean markCompacting(Collection sstables, boolean
newTables, boolean offline)
{
assert sstables != null && !Iterables.isEmpty(sstables);
while (true)
{
final View currentView = view.get();
if (Iterables.any(sstables, Predicates.in(currentView.compacting)))
return false;
Predicate live = new Predicate()
{
public boolean apply(SSTableReader sstable)
{
{color:#FF} *//If the sstable instance in sstablesMap is not the one we
are trying to mark as compacting, it will return false, and markCompacting will
return //false,*{color}*{color:#FF}and fall into infinite loop
eventually.{color}*
return{color:#FF} currentView.sstablesMap.get(sstable) == sstable{color}
&& !sstable.isMarkedCompacted();
}
};
if (newTables)
assert !Iterables.any(sstables, Predicates.in(currentView.sstables));
else if (!offline && !Iterables.all(sstables, live))
return false;
View newView = currentView.markCompacting(sstables);
if (view.compareAndSet(currentView, newView))
{
notifyCompacting(sstables, true);
return true;
}
}
}
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
>
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703482#comment-16703482
]
Ariel Weisberg commented on CASSANDRA-14908:
There is CASSANDRA-14079 which sort of works around this, but it was only
merged to 3.11 and 4.0. CASSANDRA-13948 is another example of an infinite loop
bug. I'm not sure what we would do in 2.1 at this point.
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0
> tid=0x7f5a3d4ba000 nid=0x24063 waiting for monitor entry
> [0x7fa8337bd000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
>
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703449#comment-16703449
]
Ariel Weisberg commented on CASSANDRA-14908:
There has been stuff like CASSANDRA-11172. I wasn't thinking of anything
concrete. From the thread names it maybe looks like it's the same compaction
task still running in both sets of stacks.
For instance [this
loop|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java#L121]
has the potential to iterate indefinitely if it can't mark the tables
compacting.
@krummas WDYT?
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702600#comment-16702600
]
wu taiyin commented on CASSANDRA-14908:
---
Thank you very much for you reply first. Yes, The attachment stack log is just
a fragment of the complete stack，and I update the two entire stack logs at
different time as attchment.
I get different stack log at different time , all the stack log are similar.
These stack logs show the function “getNextBackgroundTask” get the
lock"0x7faa776209b0 and never return, and dont release the lock
0x7faa776209b0. So the status of other thread which want to acquire
"0x7faa776209b0" is "BLOCKED".
All the flush task pending because the stage of MemtableFlushWriter need to
acquire lock"0x7faa776209b0 also , and the mutation request to this node
failed.
I noticed that you have met the infinite loops in compaction strategies before,
what'is the reason?
wish your reply。
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: deadlock stack.txt
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702330#comment-16702330
]
Ariel Weisberg commented on CASSANDRA-14908:
This doesn't look like a deadlock as presented?
3 threads hold no locks and are waiting to acquire 0x7faa776209b0
One thread holds 0x7faa776209b0 and at least from the stack trace might be
making progress. It is busy getting compaction candidates.
I am sure what you are saying about the system not making progress is true it's
just that I can't tell immediately from the stacks you presented why that might
be. It seems like those other threads should make progress eventually unless
{noformat}
at
org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
{noformat}
never returns. Which could be possible. Infinite loops in compaction strategies
have occurred before.
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: deadlock stack.txt
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
[jira] [Commented] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[
https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696298#comment-16696298
]
wu taiyin commented on CASSANDRA-14908:
---
All wirte request to this node failed because all flush task is pending due to
deadlock.
> Deadlock occurs when executing a file selection in levelcompact
> ---
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
>Reporter: wu taiyin
>Priority: Major
> Attachments: deadlock stack.txt
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4
> tid=0x7f73244cc000 nid=0x1919a waiting for monitor entry
> [0x7fa94e13b000]
> java.lang.Thread.State: {color:#FF}BLOCKED{color} (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at
> org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4
> tid=0x7f7324451800 nid=0x19199 runnable [0x7fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at
> {color:#FF}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF} - locked <0x7faa77620e70> (a
> org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF} at{color}
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x7faa77620e30> (a
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0
> tid=0x7f5a3d4ba000 nid=0x24063 waiting for monitor entry
> [0x7fa8337bd000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x7faa776209b0> (a
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
>
