did the amount of data finally exceed your per machine RAM capacity?
is it the same 20% each time you read? or do your periodic reads
eventually work through the entire dataset?
if you are essentially table scanning your data set, and the size
exceeds available RAM, then a degradation like that isn't crazy. and
this is indicated with your iowait %%
On Fri, Oct 12, 2012 at 6:33 AM, Brian Tarbox tar...@cabotresearch.com wrote:
I have a two node cluster hosting a 45 gig dataset. I periodically have to
read a high fraction (20% or so) of my 'rows', grabbing a few thousand at a
time and then processing them.
This used to result in about 300-500 reads a second which seemed quite good.
Recently that number has plummeted to 20-50 reads a second. The obvious
question is what did I change? I certainly added more databringing my
total load from 38 or so gig to 45 or so gig but its hard to imagine that
causing this problem. The shape of my data has not changed and I haven't
changed any cassandra configuration.
Running nodetool tpstats I'm for the first time ever seeing entries under
ReadStage Active and Pending which correlates with slow reads.
Running iostat I'm seeing a significant (10-50%) of iowait where I
previously never saw higher than 1-2%
I ran a full compaction on the relevant CF (which took 3.5 hours) to no
avail.
Any suggestions on where I can look next?
Thanks.
