Re: Does "nodetool repair" need to be run on each node for a given table?

2017-03-15 Thread Thakrar, Jayesh
Thank you Eric for helping out. The reason I sent the question a second time is because I did not see my question and the first reply from the usergroup. After I sent the question a second time, I got a personal flame from somebody else too and so examined my "spam" folders and that's where I

Re: Does "nodetool repair" need to be run on each node for a given table?

2017-03-15 Thread Eric Evans
On Tue, Mar 14, 2017 at 12:04 PM, daemeon reiydelle wrote: > Am I unreasonable in expecting a poster to have looked at the documentation > before posting? And that reposting the same query WITHOUT reading the > documents (when pointed out to them) when asked to do so is not

Re: Does "nodetool repair" need to be run on each node for a given table?

2017-03-14 Thread Max Campos
ache.org <mailto:user@cassandra.apache.org>" > <user@cassandra.apache.org <mailto:user@cassandra.apache.org>> > Subject: RE: Does "nodetool repair" need to be run on each node for a given > table? > > Yes, whichever node initiates “nodetool repair” becomes the co

Re: Does "nodetool repair" need to be run on each node for a given table?

2017-03-14 Thread daemeon reiydelle
ts", etc, the > information is for that specific node. > > While for some other commands like "status", the info is for the whole > cluster. > > > > So is "nodetool repair" that operates at a single node level (i.e. repairs > the partitions conta

Does "nodetool repair" need to be run on each node for a given table?

2017-03-14 Thread Thakrar, Jayesh
I understand that the nodetool command connects to a specific server and for many of the commands, e.g. "info", "compactionstats", etc, the information is for that specific node. While for some other commands like "status", the info is for the whole cl

Re: Does "nodetool repair" need to be run on each node for a given table?

2017-03-13 Thread daemeon reiydelle
hile for some other commands like "status", the info is for the whole > cluster. > > > > So is "nodetool repair" that operates at a single node level (i.e. repairs > the partitions contained on the target node?). > > If so, what is the recommended approac

Does "nodetool repair" need to be run on each node for a given table?

2017-03-13 Thread Thakrar, Jayesh
I understand that the nodetool command connects to a specific server and for many of the commands, e.g. "info", "compactionstats", etc, the information is for that specific node. While for some other commands like "status", the info is for the whole cl

nodetool repair of large partition

2017-01-30 Thread Jimmy Lin
hi, if i have a row in a table that contain large data (not necessary super wide row), say 10 G and a replication factor of 3. During a repair, if the data of the row in each of the node is simply off by 1 byte, is cassandra smart enough to stream only partial of the data (maybe based on a range

Nodetool Repair Best Practices

2016-11-21 Thread Daniel Subak
Hey everyone, We've just migrated to a new Cassandra cluster running 3.7 and wanted to get some information on best practices when running nodetool repair; our last cluster was 1.2 and per the documentation it seems that a lot of behavior has changed between those versions. >From a r

strange node load decrease after nodetool repair -pr

2016-10-20 Thread Oleg Krayushkin
Hi. After I've run token-ranged repair from node at 12.5.13.125 with nodetool repair -full -st ${start_tokens[i]} -et ${end_tokens[i]} on every token range, I got this node load: -- Address Load Tokens Owns Rack UN 12.5.13.141 23.94 GB 256 32.3% rack1 DN 12.5.13.125

Re: Nodetool repair

2016-09-29 Thread Li, Guangxing
Romain, I was trying what you mentioned as below: a. nodetool stop VALIDATION b. echo run -b org.apache.cassandra.db:type=StorageService forceTerminateAllRepairSessions | java -jar /tmp/jmxterm/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7199 to stop a seemingly forever-going repair but seeing

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Alexander Dejanovski
, I'd suggest the following : >>> >>>- run nodetool tpstats on all nodes in search for running/pending >>>repair sessions >>>- If you have some, and to be sure you will avoid conflicts, roll >>>restart your cluster (all nodes) >>>

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Atul Saroha
ised to Robert yesterday, and if you want to keep on running >> incremental repair, I'd suggest the following : >> >>- run nodetool tpstats on all nodes in search for running/pending >>repair sessions >>- If you have some, and to be sure you will avoid conflicts,

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Atul Saroha
> >- run nodetool tpstats on all nodes in search for running/pending >repair sessions >- If you have some, and to be sure you will avoid conflicts, roll >restart your cluster (all nodes) >- Then, run "nodetool repair" on one node. >

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Alexander Dejanovski
nodes) - Then, run "nodetool repair" on one node. - When repair has finished on this node (track messages in the log and nodetool tpstats), check if other nodes are running anticompactions - If so, wait until they are over - If not, move on to the other node You should be a

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Atul Saroha
Hi, We are not sure whether this issue is linked to that node or not. Our application does frequent delete and insert. May be our approach is not correct for nodetool repair. Yes, we generally fire repair on all boxes at same time. Till now, it was manual with default configuration ( command

Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Alexander Dejanovski
, command line or tool, etc...) ? Thanks, On Thu, Sep 29, 2016 at 10:40 AM Atul Saroha <atul.sar...@snapdeal.com> wrote: > Hi, > > We have seen a weird behaviour in cassandra 3.6. > Once our node was went down more than 10 hrs. After that, we had ran > Nodetool repair multiple

[cassandra 3.6.] Nodetool Repair + tombstone behaviour

2016-09-29 Thread Atul Saroha
Hi, We have seen a weird behaviour in cassandra 3.6. Once our node was went down more than 10 hrs. After that, we had ran Nodetool repair multiple times. But tombstone are not getting sync properly over the cluster. On day- today basis, on expiry of every grace period, deleted records start

Re: Nodetool repair

2016-09-23 Thread Romain Hardouin
OK. If you still have issues after setting streaming_socket_timeout_in_ms != 0, consider increasing request_timeout_in_ms to a high value, say 1 or 2 minutes. See comments in https://issues.apache.org/jira/browse/CASSANDRA-7904Regarding 2.1, be sure to test incremental repair on your data

Re: Nodetool repair

2016-09-22 Thread Li, Guangxing
Thanks a lot, guys. That is lots of useful info to digest. In my cassandra.ymal, request_timeout_in_ms is set to 1, streaming_socket_timeout_in_ms is not set hence takes default of 0. Looks like 2.1x has made quite some improvement on this area. Besides, I can use incremental repair. So for

Re: Nodetool repair

2016-09-22 Thread Romain Hardouin
Alain, you replied faster, I didn't see your answer :-D

Re: Nodetool repair

2016-09-22 Thread Alain RODRIGUEZ
As Matija mentioned, my coworker Alexander worked on Reaper. I believe the branches of most interest would be: Incremental repairs on Reaper: https://github.com/adejanovski/cassandra-reaper/tree/inc-repair-that-works UI integration with incremental repairs on Reaper:

Re: Nodetool repair

2016-09-22 Thread Li, Guangxing
Romain, I had another repair that seems to just hang last night. When I did 'nodetool tpstats' on nodes, I see the following in the node where I initiated the repair: AntiEntropySessions 1 1 On all other nodes, I see: AntiEntropySessions 0 0 When I

Re: Nodetool repair

2016-09-22 Thread Romain Hardouin
I meant that pending (and active) AntiEntropySessions are a simple way to check if a repair is still running on a cluster. Also have a look at Cassandra reaper: - https://github.com/spotify/cassandra-reaper - https://github.com/spodkowinski/cassandra-reaper-ui Best, Romain Le Mercredi 21

Re: Nodetool repair

2016-09-21 Thread Li, Guangxing
Romain, I started running a new repair. If I see such behavior again, I will try what you mentioned. Thanks. On Wed, Sep 21, 2016 at 9:51 AM, Romain Hardouin wrote: > Do you see any pending AntiEntropySessions (not AntiEntropyStage) with > nodetool tpstats on nodes? > >

Re: Nodetool repair

2016-09-21 Thread Romain Hardouin
Do you see any pending AntiEntropySessions (not AntiEntropyStage) with nodetool tpstats on nodes? Romain Le Mercredi 21 septembre 2016 16h45, "Li, Guangxing" a écrit : Alain, my script actually grep through all the log files, including those system.log.*.

Re: Nodetool repair

2016-09-21 Thread Li, Guangxing
Alain, my script actually grep through all the log files, including those system.log.*. So it was probably due to a failed session. So now my script assumes the repair has finished (possibly due to failure) if it does not see any more repair related logs after 2 hours. Thanks. George. On Wed,

Re: Nodetool repair

2016-09-21 Thread Alain RODRIGUEZ
Hi George, That's the best way to monitor repairs "out of the box" I could think of. When you're not seeing 2048 (in your case), it might be due to log rotation or to a session failure. Have you had a look at repair failures? I am wondering why the implementor did not put something in the log

Re: Nodetool repair

2016-09-20 Thread Li, Guangxing
Hi, I am using version 2.0.9. I have been looking into the logs to see if a repair is finished. Each time a repair is started on a node, I am seeing log line like "INFO [Thread-112920] 2016-09-16 19:00:43,805 StorageService.java (line 2646) Starting repair command #41, repairing 2048 ranges for

Re: Nodetool repair

2016-09-20 Thread Jens Rantil
On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ wrote: ... > - The size of your data > - The number of vnodes > - The compaction throughput > - The streaming throughput > - The hardware available > - The load of the cluster > - ... > I've also heard that the number of

Re: Nodetool repair

2016-09-19 Thread Alain RODRIGUEZ
| grep -v 100%' You should think about what would be a good repair strategy according to your use case and workload (run repairs by night ? Use subranges ?). Keep in mind that "nodetool repair" is useful to reduce entropy in your cluster, and so reducing the risk of inconsistencie

Re: Nodetool repair

2016-09-19 Thread Jens Rantil
Hi Lokesh, Which version of Cassandra are you using? Which compaction strategy are you using? AFAIK, a repair doesn't trigger a major compaction, but I might be wrong here. What you could do is to run a repair for a subset of the ring (see `-st` and `-et` `nodetool repair` parameters). If you

Nodetool repair

2016-09-18 Thread Lokesh Shrivastava
Hi, I tried to run nodetool repair command on one of my keyspaces and found that it took lot more time than I anticipated. Is there a way to know in advance the ETA of manual repair before triggering it? I believe repair performs following operations - 1) Major compaction 2) Exchange of merkle

Re: nodetool repair uses option '-local' and '-pr' togather

2016-09-05 Thread Paulo Motta
016-09-01 14:28 GMT-03:00 Li, Guangxing <guangxing...@pearson.com>: >> >>> Thanks for the info, Paulo. >>> >>> My cluster is in AWS, the keyspace has replication factor 3 with >>> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b

Re: nodetool repair uses option '-local' and '-pr' togather

2016-09-05 Thread Christopher Bradford
the node in us-east-1d has all the (data) replicas, right? If so, if >> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY >> on the node in us-east-1d? In other words, does 'nodetool repair' started >> on node in us-east-1d also cause repairs on replicas

Re: nodetool repair uses option '-local' and '-pr' togather

2016-09-02 Thread Paulo Motta
, would it be enough to run 'nodetool repair' ONLY > on the node in us-east-1d? In other words, does 'nodetool repair' started > on node in us-east-1d also cause repairs on replicas on other nodes? I am > seeing different answers in discussion like this http://dba.stackexchange. > com/ques

Re: nodetool repair uses option '-local' and '-pr' togather

2016-09-01 Thread Li, Guangxing
, right? If so, if I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY on the node in us-east-1d? In other words, does 'nodetool repair' started on node in us-east-1d also cause repairs on replicas on other nodes? I am seeing different answers in discussion like this http

Re: nodetool repair uses option '-local' and '-pr' togather

2016-09-01 Thread Paulo Motta
https://issues.apache.org/jira/browse/CASSANDRA-7450 2016-09-01 13:11 GMT-03:00 Li, Guangxing <guangxing...@pearson.com>: > Hi, > > I have a cluster running 2.0.9 with 2 data centers. I noticed that > 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the

nodetool repair uses option '-local' and '-pr' togather

2016-09-01 Thread Li, Guangxing
Hi, I have a cluster running 2.0.9 with 2 data centers. I noticed that 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the node's data size is 39 GB and the largest SSTable size is like 7 GB so the column family is not huge, SizeTieredCompactionStrategy is used). Repairing

Re: nodetool repair with -pr and -dc

2016-08-19 Thread Jérôme Mainaud
epair command with both -pr and -local rejected on an 2.2.6 > cluster. > The exact command was : nodetool repair --full -par -pr -local -j 4 > > The message is “You need to run primary range repair on all nodes in the > cluster”. > > Reading the code and previously cited CASSA

Re: nodetool repair with -pr and -dc

2016-08-19 Thread Romain Hardouin
ommand with both -pr and -local rejected on an 2.2.6 cluster. The exact command was : nodetool repair --full -par -pr -local -j 4 The message is  “You need to run primary range repair on all nodes in the cluster”. Reading the code and previously cited CASSANDRA-7450, it should have been accepted

Re: nodetool repair with -pr and -dc

2016-08-19 Thread Jérôme Mainaud
Hello, I've got a repair command with both -pr and -local rejected on an 2.2.6 cluster. The exact command was : nodetool repair --full -par -pr -local -j 4 The message is “You need to run primary range repair on all nodes in the cluster”. Reading the code and previously cited CASSANDRA-7450

Re: nodetool repair with -pr and -dc

2016-08-11 Thread kurt Greaves
-D does not do what you think it does. I've quoted the relevant documentation from the README: > > Multiple > Datacenters > > If you have multiple datacenters in your ring, then you MUST specify the > name of the

Re: nodetool repair with -pr and -dc

2016-08-11 Thread Paulo Motta
> if we want to use -pr option ( which i suppose we should to prevent duplicate checks) in 2.0 then if we run the repair on all nodes in a single DC then it should be sufficient and we should not need to run it on all nodes across DC's? No, because the primary ranges of the nodes in other DCs

Re: nodetool repair with -pr and -dc

2016-08-10 Thread Anishek Agarwal
ok thanks, so if we want to use -pr option ( which i suppose we should to prevent duplicate checks) in 2.0 then if we run the repair on all nodes in a single DC then it should be sufficient and we should not need to run it on all nodes across DC's ? On Wed, Aug 10, 2016 at 5:01 PM, Paulo Motta

Re: nodetool repair with -pr and -dc

2016-08-10 Thread Paulo Motta
On 2.0 repair -pr option is not supported together with -local, -hosts or -dc, since it assumes you need to repair all nodes in all DCs and it will throw and error if you try to run with nodetool, so perhaps there's something wrong with range_repair options parsing. On 2.1 it was added support to

nodetool repair with -pr and -dc

2016-08-10 Thread Anishek Agarwal
Hello, We have 2.0.17 cassandra cluster(*DC1*) with a cross dc setup with a smaller cluster(*DC2*). After reading various blogs about scheduling/running repairs looks like its good to run it with the following -pr for primary range only -st -et for sub ranges -par for parallel -dc to make sure

Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3

2016-06-22 Thread Bhuvan Rawal
>>>> >>>> — >>>> Robert Stupp >>>> @snazy >>>> >>>> On 20 Jun 2016, at 18:13, Bhuvan Rawal <bhu1ra...@gmail.com> wrote: >>>> >>>> Hi All, >>>> >>>> We are running Cassand

Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3

2016-06-22 Thread Marcus Eriksson
t Stupp >>> @snazy >>> >>> On 20 Jun 2016, at 18:13, Bhuvan Rawal <bhu1ra...@gmail.com> wrote: >>> >>> Hi All, >>> >>> We are running Cassandra 3.0.3 on Production with Max Heap Size of 8GB. >>> There has been a consis

Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3

2016-06-22 Thread Bhuvan Rawal
t; >> — >> Robert Stupp >> @snazy >> >> On 20 Jun 2016, at 18:13, Bhuvan Rawal <bhu1ra...@gmail.com> wrote: >> >> Hi All, >> >> We are running Cassandra 3.0.3 on Production with Max Heap Size of 8GB. >> There has been a consistent

Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3

2016-06-20 Thread Atul Saroha
>> Hi All, >> >> We are running Cassandra 3.0.3 on Production with Max Heap Size of 8GB. >> There has been a consistent issue with nodetool repair for a while and >> we have tried issuing it with multiple options --pr, --local as well, >> sometimes node went down

Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3

2016-06-20 Thread Paulo Motta
except upgrading to 3.7 as > that change went into 3.6 and not into 3.0.x. > > — > Robert Stupp > @snazy > > On 20 Jun 2016, at 18:13, Bhuvan Rawal <bhu1ra...@gmail.com> wrote: > > Hi All, > > We are running Cassandra 3.0.3 on Production with Max Heap Size o

Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3

2016-06-20 Thread Robert Stupp
nning Cassandra 3.0.3 on Production with Max Heap Size of 8GB. There > has been a consistent issue with nodetool repair for a while and we have > tried issuing it with multiple options --pr, --local as well, sometimes node > went down with Out of Memory error and at times nodes did stopped conn

Re: Nodetool repair inconsistencies

2016-06-08 Thread Jason Kania
the sstables in a directory on one machine, run 'nodetool repair' followed by 'nodetool compact', that directory remains empty. My understanding has been that these equivalently named directories should contain roughly the same amount of content. Thanks, Jason From: Paul Fife <paulf...@gmail.

Re: Nodetool repair inconsistencies

2016-06-08 Thread Paul Fife
.ka...@ymail.com> wrote: > Hi Romain, > > The problem is that there is no error to share. I am focusing on the > inconsistency that when I run nodetool repair, get no errors and yet the > content in the same directory on the different nodes is vastly different. > This lack of

Re: Nodetool repair inconsistencies

2016-06-08 Thread Jason Kania
Hi Romain, The problem is that there is no error to share. I am focusing on the inconsistency that when I run nodetool repair, get no errors and yet the content in the same directory on the different nodes is vastly different. This lack of an error is nature of my question, not the nodetool

Re: Nodetool repair inconsistencies

2016-06-08 Thread Romain Hardouin
ng a 3 node cluster of 3.0.6 instances and encountered an error when running nodetool compact. I then ran nodetool repair. No errors were returned. I then attempted to run nodetool compact again, but received the same error so the repair made no correction and reported no errors. After that, I

Nodetool repair inconsistencies

2016-06-07 Thread Jason Kania
I am running a 3 node cluster of 3.0.6 instances and encountered an error when running nodetool compact. I then ran nodetool repair. No errors were returned. I then attempted to run nodetool compact again, but received the same error so the repair made no correction and reported no errors. After

Re: [C*3.0.3]lucene indexes not deleted and nodetool repair makes DC unavailable

2016-05-17 Thread Andres de la Peña
Hi Siddarth, Lucene doesn't immediately remove deleted documents from disk. Instead, it just marks them as deleted, and they are effectively removed during segments merge. This is quite similar to how C* manages deletions with tombstones and compactions. Regards, 2016-05-17 17:30 GMT+01:00

Re: Nodetool repair question

2016-05-10 Thread Joel Knighton
No - repair does not change token ownership. The up/down state of a node is not related to token ownership. On Tue, May 10, 2016 at 3:26 PM, Anubhav Kale wrote: > Hello, > > > > Suppose I have 3 nodes, and stop Cassandra on one of them. Then I run a > repair. Will

Nodetool repair question

2016-05-10 Thread Anubhav Kale
Hello, Suppose I have 3 nodes, and stop Cassandra on one of them. Then I run a repair. Will repair move the token ranges from down node to other node ? In other words in any situation, does repair operation ever change token ownership ? Thanks !

Re: [C*3.0.3]lucene indexes not deleted and nodetool repair makes DC unavailable

2016-05-10 Thread Eduardo Alonso
responsive during repair. You can try several ways to avoid this: A.- Run nodetool repair -pr in every node one at a time. You'll divide the repair in three slots. B.- Maybe your use case fits in incremental repairs Cheers Eduardo Alonso Vía de las dos Castillas, 33, Ática 4, 3ª

Re: [C*3.0.3]lucene indexes not deleted and nodetool repair makes DC unavailable

2016-05-09 Thread Siddharth Verma
Hi Eduardo, Thanks for your help on stratio index problem As per your questions. 1. We ran nodetool repair on one box(no range repair), but due to it, entire DC was non responsive. It was up, but we were not able to connect. 2. RF is 3, and we have 2 DCs each with 3 nodes. 3. Consistency level

RE: [C*3.0.3]lucene indexes not deleted and nodetool repair makes DC unavailable

2016-05-06 Thread Jacques-Henri Berthemet
@cassandra.apache.org Subject: Re: [C*3.0.3]lucene indexes not deleted and nodetool repair makes DC unavailable Hi Siddaharth: I have tested with apache cassandra 3.0.3 and cassandra-lucene-index-3.0.3.1 and works well but with dsc-cassandra-3.0.3 it does not delete the lucene files. Please, can you open

Re: [C*3.0.3]lucene indexes not deleted and nodetool repair makes DC unavailable

2016-05-06 Thread Eduardo Alonso
second question, depending on the consistency level used for writes the repairs in a DC could be very resource-intensive, I have some questions: *1º- How do you run nodetool repair*? Do you run it in every machine at the same time or waits for finishing in one machine to run it in the next? Do you use p

[C*3.0.3]lucene indexes not deleted and nodetool repair makes DC unavailable

2016-05-06 Thread Siddharth Verma
Hi, I have 2 queries. We are using cassandra dsc 3.0.3 and stratio lucene indexes on tables. 1. when table is truncated, lucene index is not cleared for the same. we see that it still occupied space on disk. 2. when we run nodetool repair, all node are up (nodetool status) but we can't connect

Re: ClosedChannelExcption while nodetool repair

2016-01-12 Thread Paulo Motta
gt; I have some problems recently on my cassandra cluster. I am running 12 > nodes with 2.2.4 and while repairing with a plain "nodetool repair". In > system.log I can find > > ERROR [STREAM-IN-/172.17.2.233] 2016-01-08 08:32:38,327 > StreamSession.java:524 - [Stream #5f96e8b0-

ClosedChannelExcption while nodetool repair

2016-01-12 Thread Jan Kesten
Hi, I have some problems recently on my cassandra cluster. I am running 12 nodes with 2.2.4 and while repairing with a plain "nodetool repair". In system.log I can find ERROR [STREAM-IN-/172.17.2.233] 2016-01-08 08:32:38,327 StreamSession.java:524 - [Stream #5f96e8b0-b5e2-11e5-b4da-43

RE: management and monitoring nodetool repair

2015-10-19 Thread aeljami.ext
Thx Carlos, How can I get information on error during repair ? Thx. De : Carlos Alonso [mailto:i...@mrcalonso.com] Envoyé : lundi 19 octobre 2015 11:09 À : user@cassandra.apache.org Objet : Re: management and monitoring nodetool repair So repair process has two phases: First one is all about

Re: management and monitoring nodetool repair

2015-10-19 Thread Carlos Alonso
! Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> On 16 October 2015 at 14:09, <aeljami@orange.com> wrote: > Hi, > > I'm looking for a tool for management and monitoring of the status of > nodetool repair. > > Currently I am trying to tes

Re: management and monitoring nodetool repair

2015-10-19 Thread Carlos Alonso
Alonso [mailto:i...@mrcalonso.com] > *Envoyé :* lundi 19 octobre 2015 11:09 > *À :* user@cassandra.apache.org > *Objet :* Re: management and monitoring nodetool repair > > > > So repair process has two phases: > > > > First one is all about calculating Merkel trees

management and monitoring nodetool repair

2015-10-16 Thread aeljami.ext
Hi, I'm looking for a tool for management and monitoring of the status of nodetool repair. Currently I am trying to test cassandra-reaper, but if you tested other tools thank you to share. Thanks

Re: Nodetool repair with Load times 5

2015-08-19 Thread Jean Tremblay
will be done on all nodes. So if I understand right, one should do a “nodetool repair -par -pr -inc” on all nodes one after the other? Is this correct? I have a second cluster, a smaller one, one with only three nodes. Configured exactly the same way, except that it has other seeds

Nodetool repair with Load times 5

2015-08-18 Thread Jean Tremblay
Hi, I have a phenomena I cannot explain, and I would like to understand. I’m running Cassandra 2.1.8 on a cluster of 5 nodes. I’m using replication factor 3, with most default settings. Last week I done a nodetool status which gave me on each node a load of about 200 GB. Since then there was

Re: Nodetool repair with Load times 5

2015-08-18 Thread Mark Greene
Hey Jean, Did you try running a nodetool cleanup on all your nodes, perhaps one at a time? On Tue, Aug 18, 2015 at 3:59 AM, Jean Tremblay jean.tremb...@zen-innovations.com wrote: Hi, I have a phenomena I cannot explain, and I would like to understand. I’m running Cassandra 2.1.8 on a

Re: Nodetool repair with Load times 5

2015-08-18 Thread Jean Tremblay
No. I did not try. I would like to understand what is going on before I make my problem, maybe even worse. I really would like to understand: 1) Is this normal? 2) What is the meaning of the column Load? 3) Is there anything to fix? Can I leave it like that? 4) Did I do something wrong?

Re: Nodetool repair with Load times 5

2015-08-18 Thread Alain RODRIGUEZ
Hi Jean, I might help, but I need to know what you have done recently (change the RF, Add remove node, cleanups, anything else as much as possible...) Also, could you please do the nodetool status *myks* for your keyspace(s) ? We will then be able to know the theoretical ownership of each node

DTCS - nodetool repair - TTL

2015-06-24 Thread Anishek Agarwal
Hello all, We are running c* version 2.0.15. We have 5 nodes with RF=3. We are using DTCS and on all inserts we have a TTL of 30 days. We have no deletes.We just have one CF. When i run nodetool repair on a node i notice a lot of extra sst tables created, this I think is due to the fact that its

RE: nodetool repair

2015-06-19 Thread Jens Rantil
(but should get faster as you get further through the ring). Sean Durity From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Friday, June 19, 2015 3:56 AM To: user@cassandra.apache.org Subject: Re: nodetool repair Hi, This is not necessarily true. Repair will induce compactions only if you have

Re: nodetool repair

2015-06-19 Thread Jean Tremblay
Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction, does it mean that it would also clean up my tombstone from my LeveledCompactionStrategy tables at the same time? Thanks for your help

Re: nodetool repair

2015-06-19 Thread arun sirimalla
Yes compactions will remove tombstones On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay jean.tremb...@zen-innovations.com wrote: Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction

Long nodetool repair use nodetool repair incremental

2015-06-19 Thread Jean Tremblay
the incremental nodetool repair”. That really seems to be the option for me. Few questions: 1) Why is it written in the documentation that it is suggested to do incremental repair daily, and full repair on the weekends? Is repair incremental not good enough or not safe? 2) In order to use

Re: nodetool repair

2015-06-19 Thread Alain RODRIGUEZ
nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction, does it mean that it would also clean up my tombstone from my LeveledCompactionStrategy tables at the same time? Thanks for your help. On 19 Jun 2015, at 07:56 , arun sirimalla

Re: nodetool repair

2015-06-19 Thread Jean Tremblay
...@gmail.com] Sent: Friday, June 19, 2015 3:56 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: nodetool repair Hi, This is not necessarily true. Repair will induce compactions only if you have entropy in your cluster. If not it will just read your data to compare all

RE: nodetool repair

2015-06-19 Thread SEAN_R_DURITY
get faster as you get further through the ring). Sean Durity From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Friday, June 19, 2015 3:56 AM To: user@cassandra.apache.org Subject: Re: nodetool repair Hi, This is not necessarily true. Repair will induce compactions only if you have

nodetool repair

2015-06-18 Thread Jean Tremblay
available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node

Re: nodetool repair

2015-06-18 Thread Jonathan Haddad
and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair

Re: nodetool repair

2015-06-18 Thread Jean Tremblay
Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another

Re: nodetool repair

2015-06-18 Thread arun sirimalla
Hi Jean, Running nodetool repair on a node will repair only that node in the cluster. It is recommended to run nodetool repair on one node at a time. Few things to keep in mind while running repair 1. Running repair will trigger compactions 2. Increase in CPU utilization. Run node tool

Re: How to stop nodetool repair in 2.1.2?

2015-04-16 Thread Paul Leddy
Will JMX be more readily available by 2100? On 04/15/2015 10:14 PM, Benyi Wang wrote: Using JMX worked. Thanks a lot. On Wed, Apr 15, 2015 at 3:57 PM, Robert Coli rc...@eventbrite.com mailto:rc...@eventbrite.com wrote: On Wed, Apr 15, 2015 at 3:30 PM, Benyi Wang bewang.t...@gmail.com

Re: How to stop nodetool repair in 2.1.2?

2015-04-15 Thread Sebastian Estevez
nodetool stop *VALIDATION* On Apr 15, 2015 5:16 PM, Benyi Wang bewang.t...@gmail.com wrote: I ran nodetool repair -- keyspace table for a table, and it is still running after 4 days. I knew there is an issue for repair with vnodes https://issues.apache.org/jira/browse/CASSANDRA-5220. My

Re: How to stop nodetool repair in 2.1.2?

2015-04-15 Thread Robert Coli
On Wed, Apr 15, 2015 at 3:30 PM, Benyi Wang bewang.t...@gmail.com wrote: It didn't work. I ran the command on all nodes, but I still can see the repair activities. Your input as an operator who wants a nodetool command to trivially stop repairs is welcome here :

Re: How to stop nodetool repair in 2.1.2?

2015-04-15 Thread Benyi Wang
It didn't work. I ran the command on all nodes, but I still can see the repair activities. On Wed, Apr 15, 2015 at 3:20 PM, Sebastian Estevez sebastian.este...@datastax.com wrote: nodetool stop *VALIDATION* On Apr 15, 2015 5:16 PM, Benyi Wang bewang.t...@gmail.com wrote: I ran nodetool

How to stop nodetool repair in 2.1.2?

2015-04-15 Thread Benyi Wang
I ran nodetool repair -- keyspace table for a table, and it is still running after 4 days. I knew there is an issue for repair with vnodes https://issues.apache.org/jira/browse/CASSANDRA-5220. My question is how I can kill this sequential repair? I killed the process which I ran the repair

Re: How to stop nodetool repair in 2.1.2?

2015-04-15 Thread Benyi Wang
Using JMX worked. Thanks a lot. On Wed, Apr 15, 2015 at 3:57 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Apr 15, 2015 at 3:30 PM, Benyi Wang bewang.t...@gmail.com wrote: It didn't work. I ran the command on all nodes, but I still can see the repair activities. Your input as an

Re: nodetool repair options

2015-01-23 Thread Robert Coli
repair is probably not usable at all with RF5 or so, unless you set an extremely long gc_grace_seconds. What are the recommended options for regular, periodic repair? (Snapshot/incremental repair, default IIRC in newer Cassandra, changes many of these assumptions. I refer to old-style nodetool

Re: Does nodetool repair stop the node to answer requests ?

2015-01-23 Thread Ryan Svihla
For the cases where repair brings down a cluster, I can safely say that cluster has more problems than just repair. Consider the load caused by repair to be somewhat equivalent to losing a node. This is not a 1 for 1 comparison, as the node you're running repair on is up albeit busy,and the

Re: Does nodetool repair stop the node to answer requests ?

2015-01-22 Thread SEGALIS Morgan
what do you mean by operating correctly ? I only use dynamic columns if that helps... 2015-01-22 19:10 GMT+01:00 Robert Coli rc...@eventbrite.com: On Thu, Jan 22, 2015 at 9:36 AM, SEGALIS Morgan msega...@gmail.com wrote: So I wondered, does a nodetool repair make the server stop serving

Re: Does nodetool repair stop the node to answer requests ?

2015-01-22 Thread SEGALIS Morgan
, just need to wrap my head around the network topology and understand what I do before starting a new node. 2015-01-22 19:15 GMT+01:00 Flavien Charlon flavien.char...@gmail.com: I don't think you can do nodetool repair on a single node cluster. Still, one day or another you'll have to reboot your

Re: Does nodetool repair stop the node to answer requests ?

2015-01-22 Thread Tim Heckman
On Thu, Jan 22, 2015 at 10:22 AM, Jan cne...@yahoo.com wrote: Running a 'nodetool repair' will 'not' bring the node down. It's not something that happens during normal operation. If something goes sideways, and the resource usage climbs, a repair can definitely cripple a node. Your question

<    1   2   3   4   5   >