We are observing increasing number of TimedOutExceptions in cassandra 0.5 trunk although the load seems fairly low (about 400 reads/writes per second). cfstats reports that operations are taking less than 2 ms on average.
2 Things I have noticed looking at the source code. 1- TimedOutExceptions are silently swallowed by Cassandra and not reported in the logs even at debug level 2- readstats does not account for these long time running queries that time out. I'm wondering, what could be causing the system to go haywire like this? and how much is a reasonable read/write performance on some of the user's successful deployments before the system starts getting bad response times. Thanks Ramzi
