Hello,
I am evaluating using cassandra and I'm running into some strange IO
behavior that I can't explain, I'd like some help/ideas to troubleshoot it.
I am running a 1 node cluster with a keyspace consisting of two columns
families, one of which has dozens of supercolumns itself containing dozens
of columns.
All in all, this is a couple gigabytes of data, 12GB on the hard drive.
The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM and
an i5 processor (4 cores).
Keyspace: xxxxxxxxxxxxxxxxxxx
Read Count: 460754852
Read Latency: 1.108205793092766 ms.
Write Count: 30620665
Write Latency: 0.01411020877567486 ms.
Pending Tasks: 0
Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
SSTable count: 5
Space used (live): 548700725
Space used (total): 548700725
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 11
Read Count: 2891192
Read Latency: NaN ms.
Write Count: 3157547
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 367396
Key cache size: 367396
Key cache hit rate: NaN
Row cache capacity: 112683
Row cache size: 112683
Row cache hit rate: NaN
Compacted row minimum size: 125
Compacted row maximum size: 924
Compacted row mean size: 172
Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
SSTable count: 7
Space used (live): 8707538781
Space used (total): 8707538781
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 30
Read Count: 457863660
Read Latency: 2.381 ms.
Write Count: 27463118
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 4518387
Key cache size: 4518387
Key cache hit rate: 0.9247881700850826
Row cache capacity: 1349682
Row cache size: 1349682
Row cache hit rate: 0.39400533823415573
Compacted row minimum size: 125
Compacted row maximum size: 6866
Compacted row mean size: 165
My app makes a bunch of requests using a MultigetSuperSliceQuery for a set
of keys, typically a couple dozen at most. It also selects a subset of the
supercolumns. I am running 8 requests in parallel at most.
Two days, I ran a 1.5 hour process that basically read every key. The server
had no IOwaits and everything was humming along. However, right at the end
of the process, there was a huge spike in IOs. I didn't think much of it.
Today, after two days of inactivity, any query I run raises the IOs to 80%
utilization of the SSD drives even though I'm running the same query over
and over (no cache??)
Any ideas on how to troubleshoot this, or better, how to solve this ?
thanks
Philippe
