Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Alexander Dejanovski
Atul, our fork has been tested on 2.1 and 3.0.x clusters. I've just tested with a CCM 3.6 cluster and it worked with no issue. With Reaper, if you set incremental to false, it'll perform a full subrange repair with no anticompaction. You'll see this message in the logs : INFO [AntiEntropyStage:1

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Atul Saroha
Hi Alexander, There is compatibility issue raised with spotify/cassandra-reaper for cassandra version 3.x. Is it comaptible with 3.6 in fork thelastpickle/cassandra-reaper ? There are some suggestions mentioned by *brstgt* which we can try on our side. On Thu, Sep 29, 2016 at 5:42 PM, Atul Saroh

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Atul Saroha
Thanks Alexander. Will look into all these. On Thu, Sep 29, 2016 at 4:39 PM, Alexander Dejanovski < a...@thelastpickle.com> wrote: > Atul, > > since you're using 3.6, by default you're running incremental repair, > which doesn't like concurrency very much. > Validation errors are not occurring o

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Alexander Dejanovski
Atul, since you're using 3.6, by default you're running incremental repair, which doesn't like concurrency very much. Validation errors are not occurring on a partition or partition range base, but if you're trying to run both anticompaction and validation compaction on the same SSTable. Like adv

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Atul Saroha
Hi, We are not sure whether this issue is linked to that node or not. Our application does frequent delete and insert. May be our approach is not correct for nodetool repair. Yes, we generally fire repair on all boxes at same time. Till now, it was manual with default configuration ( command: "no

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Alexander Dejanovski
Hi Atul, could you be more specific on how you are running repair ? What's the precise command line for that, does it run on several nodes at the same time, etc... What is your gc_grace_seconds ? Do you see errors in your logs that would be linked to repairs (Validation failure or failure to creat

[cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Atul Saroha
Hi, We have seen a weird behaviour in cassandra 3.6. Once our node was went down more than 10 hrs. After that, we had ran Nodetool repair multiple times. But tombstone are not getting sync properly over the cluster. On day- today basis, on expiry of every grace period, deleted records start surfac