On Wed, Dec 16, 2009 at 12:46 PM, Ramzi Rabah <rra...@playdom.com> wrote: > We are observing increasing number of TimedOutExceptions in cassandra > 0.5 trunk although the load seems fairly low (about 400 reads/writes > per second). > cfstats reports that operations are taking less than 2 ms on average. > > 2 Things I have noticed looking at the source code. > > 1- TimedOutExceptions are silently swallowed by Cassandra and not > reported in the logs even at debug level
It's reported to the client. Hardly "swallowed" :) > 2- readstats does not account for these long time running queries that > time out. Right. But the CF-level stats do. > I'm wondering, what could be causing the system to go haywire like > this? Hard to say without more information. One shot in the dark is that get_key_range is a major offender sometimes, as well as workloads that do lots of deletes + re-inserts for the same keys. -Jonathan
