Look for log messages at the ERROR level first to find out why it's crashing.
Check for GC pressure during the repair, either using JConsole or log messages from the GCInspector. Check the nodetool tpstats to get an idea if the nodes are saturated, i.e. are their tasks in the pending list. Or are they just running with high latency. If a node crashes when calculating the Merkle tree's for it's neighbours the repair will hang (for 48 hours i think) on the node that initiated the repair. I dont think this is immediately obvious though tpstats . Start with why it's crashing and whats happening with the GC. Cheers ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 16 Jun 2011, at 10:20, Aurynn Shaw wrote: > Hey all; > > So, we have Cassandra running on a 5-server ring, with a RF of 3, and we're > regularly seeing major slowdowns in read & write performance while running > nodetool repair, as well as the occasional Cassandra crash during the repair > window - slowdowns past 10 seconds to perform a single write. > > The repair cycle runs nightly on a different server, so each server has it > run once a week. > > We're running 0.7.0 currently, and we'll be upgrading to 0.7.6 shortly. > > System load on the Cassandra servers is never more than 10% CPU and utterly > minimal IO usage, so I wouldn't think we'd be seeing issues quite like this. > > What sort of knobs should I be looking at tuning to reduce the impact that > nodetool repair has on Cassandra? What questions should I be asking as to why > Cassandra slows down to the level that it does, and what I should be > optimizing? > > Additionally, what should I be looking for in the logs when this is > happening? There's a lot in the logs, but I'm not sure what to look for. > > Cassadra is, in this instance, backing a system that supports around a > million requests a day, so not terribly heavy traffic. > > Thanks, > > Aurynn
