Re: nodetool repair failing with "Validation failed in /X.X.X.X

Hello Shalom, Someone already tried a rolling restart of Cassandra. I will probably try rebooting the OS. Repair seems to work if you do it a keyspace at a time. Thanks for your input. Rhys On Sun, May 5, 2019 at 2:14 PM shalom sagges wrote: > Hi Rhys, > > I encountered this error after

Re: nodetool repair failing with "Validation failed in /X.X.X.X

Hi Rhys, I encountered this error after adding new SSTables to a cluster and running nodetool refresh (v3.0.12). The refresh worked, but after starting repairs on the cluster, I got the "Validation failed in /X.X.X.X" error on the remote DC. A rolling restart solved the issue for me. Hope this

Re: nodetool repair -pr

It depends on your data model. -pr only repair primary range. So if there is a keyspace with replication 'DC2:3', and you run repair -pr only on all nodes of DC1, it is not going to repair token ranges corsponding to DC2. So you will have to run on each node. -Arvinder On Fri, Jun 8, 2018, 8:42

Re: nodetool repair -pr

According docs at http://cassandra.apache.org/doc/latest/operating/repair.html?highlight=single *The -pr flag will only repair the “primary” ranges on a node, so you can repair your entire cluster by running nodetool repair -pr on each node in a single datacenter.* But I saw many places, where

Re: nodetool repair -pr

I want to repair all nodes at all data centers. Example: DC1 nodeA nodeB nodeC DC2 node D node E node F If I run `nodetool repair -pr` at nodeA nodeB and nodeC, will all ranges be repaired? On Fri, 8 Jun 2018 at 17:57 Rahul Singh wrote: > From DS dox : "Do not use -pr with this option

Re: nodetool repair -pr

>From DS dox : "Do not use -pr with this option to repair only a local data >center." On Jun 8, 2018, 10:42 AM -0400, user@cassandra.apache.org, wrote: > > nodetool repair -pr

Re: Nodetool repair multiple dc

One quick question on reaper ..what data is stored in reaper_db keyspace ? And how much does it grow? Do we have to cleanup that frequently or reaper has mechnism to slef clean ? On Friday, April 13, 2018, Alexander Dejanovski wrote: > Hi Abdul, > > Reaper has been used

Re: Nodetool repair multiple dc

Hi Abdul, Reaper has been used in production for several years now, by many companies. I've seen it handling 100s of clusters and 1000s of nodes with a single Reaper process. Check the docs on cassandra-reaper.io to see which architecture matches your cluster :

Re: Nodetool repair multiple dc

Makes sense it takes a long time since it has to reconcile against replicas in all DCs. I leverage commercial tools for production clusters, but I’m pretty sure Reaper is the best open source option. Otherwise you’ll waste a lot of time trying to figure it out own your own. No need to reinvent

Re: nodetool repair and compact

I have just this been told that my first statement is inaccurate: If 'upgradesstable' is run as a routine operation, you might forget about > it and suffer consequences. 'upgradesstable' is not only doing the > compaction. I should probably have checked upgradesstable closely before making

Re: nodetool repair and compact

Hi, it will re-write this table's sstable files to current version, while > re-writing, will evit droppable tombstones (expired + gc_grace_seconds > (default 10 days) ), if partition cross different files, they will still > be kept, but most droppable tombstones gone and size reduced. > Nice

Re: nodetool repair and compact

you may use: nodetool upgradesstables -a keyspace_name table_name it will re-write this table's sstable files to current version, while re-writing, will evit droppable tombstones (expired + gc_grace_seconds (default 10 days) ), if partition cross different files, they will still be kept, but

Re: nodetool repair and compact

You’ll find the answers to your questions (and quite a bit more) in this blog post from my coworker: http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html Repair doesn’t clean up tombstones,

Re: Nodetool Repair --full

Worth noting that if you have racks == RF you only need to repair one rack to repair all the data in the cluster if you *don't* use -pr. Also note that full repairs on >=3.0 case anti-compactions and will mark things as repaired, so once you start repairs you need to keep repairing to ensure you

Re: Nodetool Repair --full

Hi Jonathan, If you want to repair just one node (for example if it has been down for more than 3h), run “nodetool repair -full” on that node. This will bring all data on that node up to date. If you want to repair all data on the cluster, run “nodetool repair -full -pr” on each node. This

Re: Nodetool repair on read only cluster

Over time the various nodes likely got slightly out of sync - dropped mutations primarily, during Long GC pauses or maybe network failures In that case, repair will make all of the data match - how long it takes depends on size of data (more data takes longer to validate), size of your

Re: Nodetool repair on read only cluster

Hi Roger, As you provide incomplete information which is so tough to analyse . But if you like to refer then please check below JIRA link to check out is it useful or not. ? https://issues.apache.org/jira/browse/CASSANDRA-6616 Thanks. On Thu, Nov 30, 2017 at 9:42 AM, Roger Warner

Re: Nodetool repair -pr

It will on 2.2 and higher, yes. Also, just want to point out that it would be worth it for you to compare how long incremental repairs take vs full repairs in your cluster. There are some problems (which are fixed in 4.0) that can cause significant overstreaming when using incremental repair.

Re: nodetool repair failure

What is your GC_GRACE_SECONDS ? What kind repair option do you use for nodetool repair on a keyspace ? Did you start the repair on one node? did you use nodetool repair -pr ? or just "nodetool repair keyspace" ? How many nodetool repair processes do you use on the nodes? On Sun, Jul 30, 2017

Re: nodetool repair failure

On 2017-07-27 21:36 (-0700), Mitch Gitman wrote: > Now, the particular symptom to which that response refers is not what I was > seeing, but the response got me thinking that perhaps the failures I was > getting were on account of attempting to run "nodetool repair >

Re: nodetool repair failure

You need check the node that failed validation to find the relevant error. The IP should be in the logs of the node you started repair on. You shouldn't run multiple repairs on the same table from multiple nodes unless you really know what you're doing and not using vnodes. The failure you are

Re: nodetool repair failure

Michael, thanks for the input. I don't think I'm going to need to upgrade to 3.11 for the sake of getting nodetool repair working for me. Instead, I have another plausible explanation and solution for my particular situation. First, I should say that disk usage proved to be a red herring. There

Re: nodetool repair failure

On 07/27/2017 12:10 PM, Mitch Gitman wrote: > I'm using Apache Cassandra 3.10. > this is a dev cluster I'm talking about. > Further insights welcome... Upgrade and see if one of the many fixes for 3.11.0 helped? https://github.com/apache/cassandra/blob/cassandra-3.11.0/CHANGES.txt#L1-L129 If

Re: nodetool repair failure

I want to add an extra data point to this thread having encountered much the same problem. I'm using Apache Cassandra 3.10. I attempted to run an incremental repair that was optimized to take advantage of some downtime where the cluster is not fielding traffic and only repair each node's primary

Re: "nodetool repair -dc"

Hi,  I have not used dc local repair specifically but generally repair syncs all local tokens of the node with other replicas (full repair) or a subset of local tokens (-pr and subrange). Full repair with - Dc option should only sync data for all the tokens present on the node where the command

Re: "nodetool repair -dc"

Hi , My Question specific to -dc option Do we need to run this on all nodes that belongs to that DC ? Or only on one of the nodes that belongs to that DC then it will repair all nodes ? On Sat, Jul 8, 2017 at 10:56 PM, Varun Gupta wrote: > I do not see the need to run

Re: "nodetool repair -dc"

I do not see the need to run repair, as long as cluster was in healthy state on adding new nodes. On Fri, Jul 7, 2017 at 8:37 AM, vasu gunja wrote: > Hi , > > I have a question regarding "nodetool repair -dc" option. recently we > added multiple nodes to one DC center, we

RE: nodetool repair failure

: nodetool repair failure It did not help much. But other issue or error I saw when I repair the keyspace was it says "Sync failed between /xx.xx.xx.93 and /xx.xx.xx.94" this was run from .91 node. On Thu, Jun 29, 2017 at 4:44 PM, Akhil Mehra <akhilme...@gmail.com<mailto:akhi

Re: nodetool repair failure

It did not help much. But other issue or error I saw when I repair the keyspace was it says "Sync failed between /xx.xx.xx.93 and /xx.xx.xx.94" this was run from .91 node. On Thu, Jun 29, 2017 at 4:44 PM, Akhil Mehra wrote: > Run the following query and see if it gives

Re: nodetool repair failure

Run the following query and see if it gives you more information: select * from system_distributed.repair_history; Also is there any additional logging on the nodes where the error is coming from. Seems to be xx.xx.xx.94 for your last run. > On 30/06/2017, at 9:43 AM, Balaji Venkatesan

Re: nodetool repair failure

The verify and scrub went without any error on the keyspace. I ran it again with trace mode and still the same issue [2017-06-29 21:37:45,578] Parsing UPDATE system_distributed.parent_repair_history SET finished_at = toTimestamp(now()), successful_ranges = {'} WHERE

Re: nodetool repair failure

Balaji, Are you repairing a specific keyspace/table? if the failure is tied to a table, try 'verify' and 'scrub' options on .91...see if you get any errors. On Thursday, June 29, 2017, 12:12:14 PM PDT, Balaji Venkatesan wrote: Thanks. I tried with trace option

Re: nodetool repair failure

Thanks. I tried with trace option and there is not much info. Here are the few log lines just before it failed. [2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to /xx.xx.xx.91 [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog [2017-06-29 19:01:54,969]

Re: nodetool repair failure

nodetool repair has a trace option nodetool repair -tr yourkeyspacename see if that provides you with additional information. Regards, Akhil > On 28/06/2017, at 2:25 AM, Balaji Venkatesan > wrote: > > > We use Apache Cassandra 3.10-13 > > On Jun 26, 2017

Re: nodetool repair failure

We use Apache Cassandra 3.10-13 On Jun 26, 2017 8:41 PM, "Michael Shuler" wrote: What version of Cassandra? -- Michael On 06/26/2017 09:53 PM, Balaji Venkatesan wrote: > Hi All, > > When I run nodetool repair on a keyspace I constantly get "Some repair > failed"

Re: nodetool repair failure

What version of Cassandra? -- Michael On 06/26/2017 09:53 PM, Balaji Venkatesan wrote: > Hi All, > > When I run nodetool repair on a keyspace I constantly get "Some repair > failed" error, there are no sufficient info to debug more. Any help? > > Here is the stacktrace > >

Re: Nodetool repair

Romain, I was trying what you mentioned as below: a. nodetool stop VALIDATION b. echo run -b org.apache.cassandra.db:type=StorageService forceTerminateAllRepairSessions | java -jar /tmp/jmxterm/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7199 to stop a seemingly forever-going repair but seeing

Re: Nodetool repair

OK. If you still have issues after setting streaming_socket_timeout_in_ms != 0, consider increasing request_timeout_in_ms to a high value, say 1 or 2 minutes. See comments in https://issues.apache.org/jira/browse/CASSANDRA-7904Regarding 2.1, be sure to test incremental repair on your data

Re: Nodetool repair

Thanks a lot, guys. That is lots of useful info to digest. In my cassandra.ymal, request_timeout_in_ms is set to 1, streaming_socket_timeout_in_ms is not set hence takes default of 0. Looks like 2.1x has made quite some improvement on this area. Besides, I can use incremental repair. So for

Re: Nodetool repair

Alain, you replied faster, I didn't see your answer :-D

Re: Nodetool repair

As Matija mentioned, my coworker Alexander worked on Reaper. I believe the branches of most interest would be: Incremental repairs on Reaper: https://github.com/adejanovski/cassandra-reaper/tree/inc-repair-that-works UI integration with incremental repairs on Reaper:

Re: Nodetool repair

Romain, I had another repair that seems to just hang last night. When I did 'nodetool tpstats' on nodes, I see the following in the node where I initiated the repair: AntiEntropySessions 1 1 On all other nodes, I see: AntiEntropySessions 0 0 When I

Re: Nodetool repair

I meant that pending (and active) AntiEntropySessions are a simple way to check if a repair is still running on a cluster. Also have a look at Cassandra reaper: - https://github.com/spotify/cassandra-reaper - https://github.com/spodkowinski/cassandra-reaper-ui Best, Romain Le Mercredi 21

Re: Nodetool repair

Romain, I started running a new repair. If I see such behavior again, I will try what you mentioned. Thanks. On Wed, Sep 21, 2016 at 9:51 AM, Romain Hardouin wrote: > Do you see any pending AntiEntropySessions (not AntiEntropyStage) with > nodetool tpstats on nodes? > >

Re: Nodetool repair

Do you see any pending AntiEntropySessions (not AntiEntropyStage) with nodetool tpstats on nodes? Romain Le Mercredi 21 septembre 2016 16h45, "Li, Guangxing" a écrit : Alain, my script actually grep through all the log files, including those system.log.*.

Re: Nodetool repair

Alain, my script actually grep through all the log files, including those system.log.*. So it was probably due to a failed session. So now my script assumes the repair has finished (possibly due to failure) if it does not see any more repair related logs after 2 hours. Thanks. George. On Wed,

Re: Nodetool repair

Hi George, That's the best way to monitor repairs "out of the box" I could think of. When you're not seeing 2048 (in your case), it might be due to log rotation or to a session failure. Have you had a look at repair failures? I am wondering why the implementor did not put something in the log

Re: Nodetool repair

Hi, I am using version 2.0.9. I have been looking into the logs to see if a repair is finished. Each time a repair is started on a node, I am seeing log line like "INFO [Thread-112920] 2016-09-16 19:00:43,805 StorageService.java (line 2646) Starting repair command #41, repairing 2048 ranges for

Re: Nodetool repair

On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ wrote: ... > - The size of your data > - The number of vnodes > - The compaction throughput > - The streaming throughput > - The hardware available > - The load of the cluster > - ... > I've also heard that the number of

Re: Nodetool repair

Hi Lokesh, Repair is a regular, very common and yet non trivial operations in Cassandra. A lot of people are struggling with it. Some good talks were done about repairs during the summit, you might want to have a look in the Datastax youtube channel in a few days :-).

Re: Nodetool repair

Hi Lokesh, Which version of Cassandra are you using? Which compaction strategy are you using? AFAIK, a repair doesn't trigger a major compaction, but I might be wrong here. What you could do is to run a repair for a subset of the ring (see `-st` and `-et` `nodetool repair` parameters). If you

Re: nodetool repair uses option '-local' and '-pr' togather

You're right Christopher, I missed the fact that with RF=3 NTS will always place a replica on us-east-1d, so in this case repair on this node would be sufficient. Thanks for clarifying! 2016-09-05 11:28 GMT-03:00 Christopher Bradford : > If each AZ has a different rack

Re: nodetool repair uses option '-local' and '-pr' togather

If each AZ has a different rack identifier and the keyspace uses NetworkTopologyStrategy with a replication factor of 3 then the single host in us-east-1d *will receive 100% of the data*. This is due to NetworkTopologyStrategy's preference for placing replicas across different racks before placing

Re: nodetool repair uses option '-local' and '-pr' togather

If I understand the way replication is done, the node in us-east-1d has all the (data) replicas, right? No, for this to be correct, you'd need to have one DC per AZ, which is not this case since you have a single DC encompassing multiple AZs. Right now, replicas will be spread in 3 distinct AZs,

Re: nodetool repair uses option '-local' and '-pr' togather

Thanks for the info, Paulo. My cluster is in AWS, the keyspace has replication factor 3 with NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in us-east-1c and 1 in us-east-1d. If I understand the way replication is done, the node in us-east-1d has all the (data) replicas,

Re: nodetool repair uses option '-local' and '-pr' togather

https://issues.apache.org/jira/browse/CASSANDRA-7450 2016-09-01 13:11 GMT-03:00 Li, Guangxing : > Hi, > > I have a cluster running 2.0.9 with 2 data centers. I noticed that > 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the > node's data size

Re: nodetool repair with -pr and -dc

Hi Romain, Thank you for your answer, I will open a ticket soon. Best -- Jérôme Mainaud jer...@mainaud.com 2016-08-19 12:16 GMT+02:00 Romain Hardouin : > Hi Jérôme, > > The code in 2.2.6 allows -local and -pr: > https://github.com/apache/cassandra/blob/cassandra-2.2. >

Re: nodetool repair with -pr and -dc

Hi Jérôme, The code in 2.2.6 allows -local and -pr:https://github.com/apache/cassandra/blob/cassandra-2.2.6/src/java/org/apache/cassandra/service/StorageService.java#L2899 But... the options validation introduced in CASSANDRA-6455 seems to break this

Re: nodetool repair with -pr and -dc

Hello, I've got a repair command with both -pr and -local rejected on an 2.2.6 cluster. The exact command was : nodetool repair --full -par -pr -local -j 4 The message is “You need to run primary range repair on all nodes in the cluster”. Reading the code and previously cited CASSANDRA-7450,

Re: nodetool repair with -pr and -dc

-D does not do what you think it does. I've quoted the relevant documentation from the README: > > Multiple > Datacenters > > If you have multiple datacenters in your ring, then you MUST specify the > name of the

Re: nodetool repair with -pr and -dc

> if we want to use -pr option ( which i suppose we should to prevent duplicate checks) in 2.0 then if we run the repair on all nodes in a single DC then it should be sufficient and we should not need to run it on all nodes across DC's? No, because the primary ranges of the nodes in other DCs

Re: nodetool repair with -pr and -dc

ok thanks, so if we want to use -pr option ( which i suppose we should to prevent duplicate checks) in 2.0 then if we run the repair on all nodes in a single DC then it should be sufficient and we should not need to run it on all nodes across DC's ? On Wed, Aug 10, 2016 at 5:01 PM, Paulo Motta

Re: nodetool repair with -pr and -dc

On 2.0 repair -pr option is not supported together with -local, -hosts or -dc, since it assumes you need to repair all nodes in all DCs and it will throw and error if you try to run with nodetool, so perhaps there's something wrong with range_repair options parsing. On 2.1 it was added support to

Re: Nodetool repair inconsistencies

com> To: user@cassandra.apache.org; Jason Kania <jason.ka...@ymail.com> Sent: Wednesday, June 8, 2016 12:55 PM Subject: Re: Nodetool repair inconsistencies Hi Jason - Did you run a major compaction after the repair completed? Do you have other reasons besides the number/size

Re: Nodetool repair inconsistencies

> Kania <jason.ka...@ymail.com> > *Sent:* Wednesday, June 8, 2016 8:30 AM > *Subject:* Re: Nodetool repair inconsistencies > > Hi Jason, > > It's difficult for the community to help you if you don't share the error > ;-) > What the logs said when you ran a major compaction? (

Re: Nodetool repair inconsistencies

compact error. Thanks, Jason From: Romain Hardouin <romainh...@yahoo.fr> To: "user@cassandra.apache.org" <user@cassandra.apache.org>; Jason Kania <jason.ka...@ymail.com> Sent: Wednesday, June 8, 2016 8:30 AM Subject: Re: Nodetool repair inconsistencies

Re: Nodetool repair inconsistencies

Hi Jason, It's difficult for the community to help you if you don't share the error ;-)What the logs said when you ran a major compaction? (i.e. the first error you encountered) Best, Romain Le Mercredi 8 juin 2016 3h34, Jason Kania a écrit : I am running a 3

Re: Nodetool repair question

No - repair does not change token ownership. The up/down state of a node is not related to token ownership. On Tue, May 10, 2016 at 3:26 PM, Anubhav Kale wrote: > Hello, > > > > Suppose I have 3 nodes, and stop Cassandra on one of them. Then I run a > repair. Will

Re: Nodetool repair with Load times 5

Dear Alain, Thanks again for your precious help. I might help, but I need to know what you have done recently (change the RF, Add remove node, cleanups, anything else as much as possible...) I have a cluster of 5 nodes all running Cassandra 2.1.8. I have a fixed schema which never changes. I

Re: Nodetool repair with Load times 5

Hey Jean, Did you try running a nodetool cleanup on all your nodes, perhaps one at a time? On Tue, Aug 18, 2015 at 3:59 AM, Jean Tremblay jean.tremb...@zen-innovations.com wrote: Hi, I have a phenomena I cannot explain, and I would like to understand. I’m running Cassandra 2.1.8 on a

Re: Nodetool repair with Load times 5

No. I did not try. I would like to understand what is going on before I make my problem, maybe even worse. I really would like to understand: 1) Is this normal? 2) What is the meaning of the column Load? 3) Is there anything to fix? Can I leave it like that? 4) Did I do something wrong?

Re: Nodetool repair with Load times 5

Hi Jean, I might help, but I need to know what you have done recently (change the RF, Add remove node, cleanups, anything else as much as possible...) Also, could you please do the nodetool status *myks* for your keyspace(s) ? We will then be able to know the theoretical ownership of each node

RE: nodetool repair

(but should get faster as you get further through the ring). Sean Durity From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Friday, June 19, 2015 3:56 AM To: user@cassandra.apache.org Subject: Re: nodetool repair Hi, This is not necessarily true. Repair will induce compactions only if you have

Re: nodetool repair

Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction, does it mean that it would also clean up my tombstone from my LeveledCompactionStrategy tables at the same time? Thanks for your help.

Re: nodetool repair

Yes compactions will remove tombstones On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay jean.tremb...@zen-innovations.com wrote: Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction,

Re: nodetool repair

Hi, This is not necessarily true. Repair will induce compactions only if you have entropy in your cluster. If not it will just read your data to compare all the replica of each piece of data (using indeed cpu and disk IO). If there is some data missing it will repair it. Though, due to merkle

Re: nodetool repair

...@gmail.com] Sent: Friday, June 19, 2015 3:56 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: nodetool repair Hi, This is not necessarily true. Repair will induce compactions only if you have entropy in your cluster. If not it will just read your data to compare all

RE: nodetool repair

get faster as you get further through the ring). Sean Durity From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Friday, June 19, 2015 3:56 AM To: user@cassandra.apache.org Subject: Re: nodetool repair Hi, This is not necessarily true. Repair will induce compactions only if you have

Re: nodetool repair

If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.com

Re: nodetool repair

Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another

Re: nodetool repair

Hi Jean, Running nodetool repair on a node will repair only that node in the cluster. It is recommended to run nodetool repair on one node at a time. Few things to keep in mind while running repair 1. Running repair will trigger compactions 2. Increase in CPU utilization. Run node tool

Re: nodetool repair options

On Fri, Jan 23, 2015 at 10:03 AM, Robert Wille rwi...@fold3.com wrote: The docs say Use -pr to repair only the first range returned by the partitioner”. What does this mean? Why would I only want to repair the first range? If you're repairing the whole cluster, repairing only the primary

Re: nodetool repair

On Fri, Jan 9, 2015 at 8:01 AM, Adil adil.cha...@gmail.com wrote: We have two DC, we are planning to schedule running nodetool repair weekly, my question is : nodetool repair is cross cluster or not? it's sufficient to run it without options on a node or should be scheduled on every node with

Re: nodetool repair exception

The official recommendation is 100k: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html I wonder if there's an advantage to this over unlimited if you're running servers which are dedicated to your Cassandra cluster (which you should be for

Re: nodetool repair exception

On Sat, Dec 6, 2014 at 8:05 AM, Eric Stevens migh...@gmail.com wrote: The official recommendation is 100k: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html I wonder if there's an advantage to this over unlimited if you're running servers

Re: nodetool repair exception

On Wed, Dec 3, 2014 at 6:37 AM, Rafał Furmański rfurman...@opera.com wrote: I see “Too many open files” exception in logs, but I’m sure that my limit is now 150k. Should I increase it? What’s the reasonable limit of open files for cassandra? Why provide any limit? ulimit allows unlimited?

Re: nodetool repair exception

As the exception indicates, nodetool just lost communication with the Cassandra node and cannot print progress any further. Check your system.log on the node, and see if your repair was completed. If there is no error, then it should be fine. On Wed, Dec 3, 2014 at 5:08 AM, Rafał Furmański

Re: nodetool repair exception

I see “Too many open files” exception in logs, but I’m sure that my limit is now 150k. Should I increase it? What’s the reasonable limit of open files for cassandra? On 3 gru 2014, at 15:02, Yuki Morishita mor.y...@gmail.com wrote: As the exception indicates, nodetool just lost communication

Re: nodetool repair stalled

Wouldn't it be a better idea to issue removenode on the crashed node, wipe the whole data directory (including system) and let it bootstrap cleanly so that it's not part of the cluster while it gets back up to speed? On Tue, Nov 11, 2014, 12:32 PM Robert Coli rc...@eventbrite.com wrote: On Tue,

Re: nodetool repair stalled

On Wed, Nov 12, 2014 at 6:50 AM, Eric Stevens migh...@gmail.com wrote: Wouldn't it be a better idea to issue removenode on the crashed node, wipe the whole data directory (including system) and let it bootstrap cleanly so that it's not part of the cluster while it gets back up to Yes, with

Re: nodetool repair stalled

Hi Eric, The data are stored in JBOD. Only one of the disk got crashed other 3 disk still holds the old data . That's why I didn't clean the whole node and issue a fresh restart Thanks Rob. Will do try that way. From: Eric Stevens Sent: ‎Wednesday‎, ‎November‎ ‎12‎, ‎2014 ‎8‎:‎21‎ ‎PM

Re: Nodetool Repair questions

Hi Vish, 1. This tool repairs inconsistencies across replicas of the row. Since latest update always wins, I dont see inconsistencies other than ones resulting from the combination of deletes, tombstones, and crashed nodes. Technically, if data is never deleted from cassandra, then nodetool

Re: Nodetool Repair questions

1. You don't have to repair if you use QUORUM consistency and you don't delete data. 2.Performance depends on size of data each node has. It's very difficult to predict. It may take days. Thank you, Andrey On Tue, Aug 12, 2014 at 2:06 PM, Viswanathan Ramachandran vish.ramachand...@gmail.com

Re: Nodetool Repair questions

Thanks Mark, Since we have replicas in each data center, addition of a new data center (and new replicas) has a performance implication on nodetool repair. I do understand that adding nodes without increasing number of replicas may improve repair performance, but in this case we are adding new

Re: Nodetool Repair questions

Andrey, QUORUM consistency and no deletes makes perfect sense. I believe we could modify that to EACH_QUORUM or QUORUM consistency and no deletes - isnt that right ? Thanks On Tue, Aug 12, 2014 at 3:10 PM, Andrey Ilinykh ailin...@gmail.com wrote: 1. You don't have to repair if you use QUORUM

Re: Nodetool Repair questions

On Tue, Aug 12, 2014 at 4:46 PM, Viswanathan Ramachandran vish.ramachand...@gmail.com wrote: Andrey, QUORUM consistency and no deletes makes perfect sense. I believe we could modify that to EACH_QUORUM or QUORUM consistency and no deletes - isnt that right? yes.

Re: nodetool repair -snapshot option?

I also expanded on a script originally written by Matt Stump @ Datastax. The readme has the reasoning behind requiring sub-range repairs. https://github.com/hancockks/cassandra_range_repair On Mon, Jun 30, 2014 at 10:20 PM, Phil Burress philburress...@gmail.com wrote: @Paulo, this is very

Re: nodetool repair saying starting and then nothing, and nothing in any of the server logs either

if the boxes are idle, you could use jstack and look at the stack… perhaps it's locked somewhere. Worth a shot. On Tue, Jul 1, 2014 at 9:24 AM, Brian Tarbox tar...@cabotresearch.com wrote: I have a six node cluster in AWS (repl:3) and recently noticed that repair was hanging. I've run with

Re: nodetool repair saying starting and then nothing, and nothing in any of the server logs either

On Tue, Jul 1, 2014 at 9:24 AM, Brian Tarbox tar...@cabotresearch.com wrote: I have a six node cluster in AWS (repl:3) and recently noticed that repair was hanging. I've run with the -pr switch. It'll do that. What version of Cassandra? =Rob

Re: nodetool repair saying starting and then nothing, and nothing in any of the server logs either

We're running 1.2.13. Any chance that doing a rolling-restart would help? Would running without the -pr improve the odds? Thanks. On Tue, Jul 1, 2014 at 1:40 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Jul 1, 2014 at 9:24 AM, Brian Tarbox tar...@cabotresearch.com wrote: I have a

  1   2   3   >