[
https://issues.apache.org/jira/browse/CASSANDRA-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom van der Woerdt updated CASSANDRA-13469:
-------------------------------------------
Reproduced In: 3.0.13
> Compaction stops; new sstables aren't recognized
> ------------------------------------------------
>
> Key: CASSANDRA-13469
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13469
> Project: Cassandra
> Issue Type: Bug
> Reporter: Tom van der Woerdt
>
> cfstats shows me this, so I know I'm in trouble :
> {code}
> SSTable count: 3419
> SSTables in each level: [2, 20/10, 154/100, 1034/1000, 0, 0,
> 0, 0, 0]
> {code}
> Disk space on these nodes (about five of them) doubled overnight, due to the
> high write volume not getting compacted. `nodetool compactionstats' shows me
> zero pending compactions.
> After a restart things are "better" :
> {noformat}
> SSTable count: 3432
> SSTables in each level: [1280/4, 21/10, 262/100, 1851/1000,
> 15, 0, 0, 0, 0]
> {noformat}
> compactionstats also shows some compactions again.
> jstack for one compaction thread (there are 24 of them, all looking the exact
> same) :
> {noformat}
> "CompactionExecutor:73017" #203194 daemon prio=1 os_prio=4
> tid=0x00007edd1e597c00 nid=0xf7e4 waiting on condition [0x00007efd16782000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000002c1859520> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$3/335359181.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Sadly I'm pretty clueless about what happened here. :(
> Oh, and tpstats :
> {noformat}
> Pool Name Active Pending Completed Blocked All
> time blocked
> MutationStage 0 0 1098466484 0
> 0
> ViewMutationStage 0 0 0 0
> 0
> ReadStage 2 0 759723051 0
> 0
> RequestResponseStage 0 0 1773795694 0
> 0
> ReadRepairStage 0 0 37372223 0
> 0
> CounterMutationStage 0 0 0 0
> 0
> MiscStage 0 0 0 0
> 0
> CompactionExecutor 0 0 336794 0
> 0
> MemtableReclaimMemory 0 0 75324 0
> 0
> PendingRangeCalculator 0 0 68 0
> 0
> GossipStage 0 0 1363346 0
> 0
> SecondaryIndexManagement 0 0 0 0
> 0
> HintsDispatcher 1 1 38 0
> 0
> MigrationStage 0 0 274693 0
> 0
> MemtablePostFlush 0 0 248358 0
> 0
> ValidationExecutor 0 0 7882 0
> 0
> Sampler 0 0 0 0
> 0
> MemtableFlushWriter 0 0 75324 0
> 0
> InternalResponseStage 0 0 259871 0
> 0
> AntiEntropyStage 0 0 20469 0
> 0
> CacheCleanupExecutor 0 0 0 0
> 0
> Native-Transport-Requests 0 0 1489435995 0
> 188985
> Message type Dropped
> READ 2
> RANGE_SLICE 0
> _TRACE 0
> HINT 0
> MUTATION 20
> COUNTER_MUTATION 0
> BATCH_STORE 0
> BATCH_REMOVE 0
> REQUEST_RESPONSE 0
> PAGED_RANGE 0
> READ_REPAIR 4
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
