Hi, I just had a weird behaviour on one of our Cassandra nodes, which I would like to share:
Short version: My pending reads went up from ~0 to the hundreds when I reduced the compactionthroughput from 16 to 2. Long version: One of our more powerful nodes had a few pending reads, while the other ones didn't. So far nothing special. Strangely neither CPU, nor IO Wait, nor disk-ops/s, nor C*-heap was particularly high. So I was wondering. That machine had two compactions and a validation(incremental) running, so I set the compactionthroughput to 2. To my surprise I saw the pending reads go up to the hundreds within 5-10 seconds. Setting the compactionthroughput back to 16 and the pending reads went back to 0 (or at least close to zero). I kept the compactionthroughput on 2 for less than a minute. So the issue is not compactions falling behind. I was able to reproduce this behaviour 5-10 times. The pending reads went up, everytime I *de*creased the compactionthroughput. I watched the pending reads while the compactionthroughput was on 16, and I never observed even a two digit pending read count while it was on compactionthroughput 16. Unfortunetaly the machine does not show this behaviour any more. Also it was only a single machine. Our setup: C* 2.2.5 with 256 vnodes + 9 nodes + incremental repair + 6GB heap My question: Did someone else ever observe such a behaviour? Is it perhaps possible that the read-path shares a lock with repair/compaction that waits on ThrottledReader while holding that lock? kind regards, Christian
