Re: Automating nodetool repair

2012-08-29 Thread aaron morton
Staggering the repairs also gives the DynamicSnitch a chance to route around nodes which maybe running slow. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 29/08/2012, at 11:19 AM, Omid Aladini wrote: >>> Secondly, what's the need for sl

Re: Automating nodetool repair

2012-08-28 Thread Omid Aladini
> > Secondly, what's the need for sleep 120? > > just give the cluster a chance to settle down between repairs... > there's no real need for it, just is there "because". Actually, repair could cause unreplicated data to be streamed and new sstables to be created. New sstables could cause pending c

Re: Automating nodetool repair

2012-08-28 Thread Aaron Turner
On Tue, Aug 28, 2012 at 1:42 PM, Edward Sargisson wrote: > Thanks a very nice approach. > > If every nodetool repair uses -pr does that satisfy the requirement to run a > repair before GCGraceSeconds expires? In otherwords, will we get a correct > result using -pr everywhere. Yep. > Secondly, wh

Re: Automating nodetool repair

2012-08-28 Thread Edward Sargisson
Thanks a very nice approach. If every nodetool repair uses -pr does that satisfy the requirement to run a repair before GCGraceSeconds expires? In otherwords, will we get a correct result using -pr everywhere. Secondly, what's the need for sleep 120? Cheers, Edward On 12-08-28 07:03 AM, Edw

Re: Automating nodetool repair

2012-08-28 Thread Mohit Agarwal
Is there any reason why cassandra doesn't do nodetool repair out of the box at some fixed intervals? On Tue, Aug 28, 2012 at 9:08 PM, Aaron Turner wrote: > Funny you mention that... i just was hearing on #cassandra this > morning that it repairs the replica set by default. I was thinking of > r

Re: Automating nodetool repair

2012-08-28 Thread Aaron Turner
Funny you mention that... i just was hearing on #cassandra this morning that it repairs the replica set by default. I was thinking of repairing every 3rd node (RF=3), but running -pr seems "cleaner". Do you know if this (repairing a replica vs node) was introduced in 1.0 or 1.1? On Tue, Aug 28,

Re: Automating nodetool repair

2012-08-28 Thread Edward Capriolo
You can consider adding -pr. When iterating through all your hosts like this. -pr means primary range, and will do less duplicated work. On Mon, Aug 27, 2012 at 8:05 PM, Aaron Turner wrote: > I use cron. On one box I just do: > > for n in node1 node2 node3 node4 ; do >nodetool -h $n repair >

Re: Automating nodetool repair

2012-08-27 Thread Aaron Turner
I use cron. On one box I just do: for n in node1 node2 node3 node4 ; do nodetool -h $n repair sleep 120 done A lot easier then managing a bunch of individual crontabs IMHO although I suppose I could of done it with puppet, but then you always have to keep an eye out that your repairs don't

Automating nodetool repair

2012-08-27 Thread Edward Sargisson
Hi all, So nodetool repair has to be run regularly on all nodes. Does anybody have any interesting strategies or tools for doing this or is everybody just setting up cron to do it? For example, one could write some Puppet code to splay the cron times around so that only one should be running