Failed decommission
This on cass 1.2.8 Ring state before decommission -- Address Load Owns Host ID TokenRack UN 10.0.0.1 38.82 GB 33.3% 21a98502-dc74-4ad0-9689-0880aa110409 1 1a UN 10.0.0.2 33.5 GB33.3% cba6b27a-4982-4f04-854d-cc73155d5f69 56713727820156407428984779325531226110 1b UN 10.0.0.3 37.41 GB 0.0% 6ba2c7d4-713e-4c14-8df8-f861fb211b0d 56713727820156407428984779325531226111 1b UN 10.0.0.4 35.7 GB33.3% bf3d4792-f3e0-4062-afe3-be292bc85ed7 11342745564031281485796955865106245 1c Trying to decommission the node ubuntu@10.0.0.3:~$ nodetool decommission Exception in thread main java.lang.NumberFormatException: For input string: 56713727820156407428984779325531226111 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:444) at java.lang.Long.parseLong(Long.java:483) at org.apache.cassandra.service.StorageService.extractExpireTime(StorageService.java:1660) at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1515) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1234) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:949) at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1116) at org.apache.cassandra.service.StorageService.leaveRing(StorageService.java:2817) at org.apache.cassandra.service.StorageService.unbootstrap(StorageService.java:2861) at org.apache.cassandra.service.StorageService.decommission(StorageService.java:2808) Now I'm in a state where the machine is still up but leaving but I can't seem to get it out of the ring. For example: % nodetool removenode 6ba2c7d4-713e-4c14-8df8-f861fb211b0d Exception in thread main java.lang.UnsupportedOperationException: Node /10.0.0.3 is alive and owns this ID. Use decommission command to remove it from the ring Any ideas? /Janne
Issue with CQLsh
Hi, I have created a column family using Cassandra-cli as: create column family default; and then inserted some record as: set default[1]['type']='bytes'; Then i tried to alter table it via cqlsh as: alter table default alter key type text; // it works alter table default alter column1 type text; // it goes for a toss surprisingly any command after that, simple hangs and i need to reset connection. Any suggestions?
Re: Failed decommission
Are you using Murmur3 or the older Random partitioner on this cluster? On Sun, Aug 25, 2013 at 3:06 AM, Janne Jalkanen janne.jalka...@ecyrd.comwrote: This on cass 1.2.8 Ring state before decommission -- Address Load Owns Host ID TokenRack UN 10.0.0.1 38.82 GB 33.3% 21a98502-dc74-4ad0-9689-0880aa110409 1 1a UN 10.0.0.2 33.5 GB33.3% cba6b27a-4982-4f04-854d-cc73155d5f69 56713727820156407428984779325531226110 1b UN 10.0.0.3 37.41 GB 0.0% 6ba2c7d4-713e-4c14-8df8-f861fb211b0d 56713727820156407428984779325531226111 1b UN 10.0.0.4 35.7 GB33.3% bf3d4792-f3e0-4062-afe3-be292bc85ed7 11342745564031281485796955865106245 1c Trying to decommission the node ubuntu@10.0.0.3:~$ nodetool decommission Exception in thread main java.lang.NumberFormatException: For input string: 56713727820156407428984779325531226111 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:444) at java.lang.Long.parseLong(Long.java:483) at org.apache.cassandra.service.StorageService.extractExpireTime(StorageService.java:1660) at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1515) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1234) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:949) at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1116) at org.apache.cassandra.service.StorageService.leaveRing(StorageService.java:2817) at org.apache.cassandra.service.StorageService.unbootstrap(StorageService.java:2861) at org.apache.cassandra.service.StorageService.decommission(StorageService.java:2808) Now I'm in a state where the machine is still up but leaving but I can't seem to get it out of the ring. For example: % nodetool removenode 6ba2c7d4-713e-4c14-8df8-f861fb211b0d Exception in thread main java.lang.UnsupportedOperationException: Node / 10.0.0.3 is alive and owns this ID. Use decommission command to remove it from the ring Any ideas? /Janne
Re: Failed decommission
Janne, We ran into this too. Appears it's a bug in 1.2.8 that is fixed in the upcoming 1.2.9. I added the steps I took to finally remove the node here: https://issues.apache.org/jira/browse/CASSANDRA-5857?focusedCommentId=13748998page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13748998 Cheers, Mike On Sun, Aug 25, 2013 at 4:06 AM, Janne Jalkanen janne.jalka...@ecyrd.comwrote: This on cass 1.2.8 Ring state before decommission -- Address Load Owns Host ID TokenRack UN 10.0.0.1 38.82 GB 33.3% 21a98502-dc74-4ad0-9689-0880aa110409 1 1a UN 10.0.0.2 33.5 GB33.3% cba6b27a-4982-4f04-854d-cc73155d5f69 56713727820156407428984779325531226110 1b UN 10.0.0.3 37.41 GB 0.0% 6ba2c7d4-713e-4c14-8df8-f861fb211b0d 56713727820156407428984779325531226111 1b UN 10.0.0.4 35.7 GB33.3% bf3d4792-f3e0-4062-afe3-be292bc85ed7 11342745564031281485796955865106245 1c Trying to decommission the node ubuntu@10.0.0.3:~$ nodetool decommission Exception in thread main java.lang.NumberFormatException: For input string: 56713727820156407428984779325531226111 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:444) at java.lang.Long.parseLong(Long.java:483) at org.apache.cassandra.service.StorageService.extractExpireTime(StorageService.java:1660) at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1515) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1234) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:949) at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1116) at org.apache.cassandra.service.StorageService.leaveRing(StorageService.java:2817) at org.apache.cassandra.service.StorageService.unbootstrap(StorageService.java:2861) at org.apache.cassandra.service.StorageService.decommission(StorageService.java:2808) Now I'm in a state where the machine is still up but leaving but I can't seem to get it out of the ring. For example: % nodetool removenode 6ba2c7d4-713e-4c14-8df8-f861fb211b0d Exception in thread main java.lang.UnsupportedOperationException: Node / 10.0.0.3 is alive and owns this ID. Use decommission command to remove it from the ring Any ideas? /Janne -- Mike Heffner m...@librato.com Librato, Inc.
Re: Failed decommission
We ran into a similar issue as well. I believe we removed the node via cqlsh from the system keyspace, restarted the cluster, then ran a repair. I'm not sure how safe this really is though. On Aug 25, 2013, at 8:47 AM, Mike Heffner m...@librato.com wrote: Janne, We ran into this too. Appears it's a bug in 1.2.8 that is fixed in the upcoming 1.2.9. I added the steps I took to finally remove the node here: https://issues.apache.org/jira/browse/CASSANDRA-5857?focusedCommentId=13748998page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13748998 Cheers, Mike On Sun, Aug 25, 2013 at 4:06 AM, Janne Jalkanen janne.jalka...@ecyrd.com wrote: This on cass 1.2.8 Ring state before decommission -- Address Load Owns Host ID TokenRack UN 10.0.0.1 38.82 GB 33.3% 21a98502-dc74-4ad0-9689-0880aa110409 1 1a UN 10.0.0.2 33.5 GB33.3% cba6b27a-4982-4f04-854d-cc73155d5f69 56713727820156407428984779325531226110 1b UN 10.0.0.3 37.41 GB 0.0% 6ba2c7d4-713e-4c14-8df8-f861fb211b0d 56713727820156407428984779325531226111 1b UN 10.0.0.4 35.7 GB33.3% bf3d4792-f3e0-4062-afe3-be292bc85ed7 11342745564031281485796955865106245 1c Trying to decommission the node ubuntu@10.0.0.3:~$ nodetool decommission Exception in thread main java.lang.NumberFormatException: For input string: 56713727820156407428984779325531226111 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:444) at java.lang.Long.parseLong(Long.java:483) at org.apache.cassandra.service.StorageService.extractExpireTime(StorageService.java:1660) at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1515) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1234) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:949) at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1116) at org.apache.cassandra.service.StorageService.leaveRing(StorageService.java:2817) at org.apache.cassandra.service.StorageService.unbootstrap(StorageService.java:2861) at org.apache.cassandra.service.StorageService.decommission(StorageService.java:2808) Now I'm in a state where the machine is still up but leaving but I can't seem to get it out of the ring. For example: % nodetool removenode 6ba2c7d4-713e-4c14-8df8-f861fb211b0d Exception in thread main java.lang.UnsupportedOperationException: Node /10.0.0.3 is alive and owns this ID. Use decommission command to remove it from the ring Any ideas? /Janne -- Mike Heffner m...@librato.com Librato, Inc.
Re: Failed decommission
Thanks; this worked for me too. /Janne On Aug 25, 2013, at 18:47 , Mike Heffner m...@librato.com wrote: Janne, We ran into this too. Appears it's a bug in 1.2.8 that is fixed in the upcoming 1.2.9. I added the steps I took to finally remove the node here: https://issues.apache.org/jira/browse/CASSANDRA-5857?focusedCommentId=13748998page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13748998 Cheers, Mike On Sun, Aug 25, 2013 at 4:06 AM, Janne Jalkanen janne.jalka...@ecyrd.com wrote: This on cass 1.2.8 Ring state before decommission -- Address Load Owns Host ID TokenRack UN 10.0.0.1 38.82 GB 33.3% 21a98502-dc74-4ad0-9689-0880aa110409 1 1a UN 10.0.0.2 33.5 GB33.3% cba6b27a-4982-4f04-854d-cc73155d5f69 56713727820156407428984779325531226110 1b UN 10.0.0.3 37.41 GB 0.0% 6ba2c7d4-713e-4c14-8df8-f861fb211b0d 56713727820156407428984779325531226111 1b UN 10.0.0.4 35.7 GB33.3% bf3d4792-f3e0-4062-afe3-be292bc85ed7 11342745564031281485796955865106245 1c Trying to decommission the node ubuntu@10.0.0.3:~$ nodetool decommission Exception in thread main java.lang.NumberFormatException: For input string: 56713727820156407428984779325531226111 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:444) at java.lang.Long.parseLong(Long.java:483) at org.apache.cassandra.service.StorageService.extractExpireTime(StorageService.java:1660) at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1515) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1234) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:949) at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1116) at org.apache.cassandra.service.StorageService.leaveRing(StorageService.java:2817) at org.apache.cassandra.service.StorageService.unbootstrap(StorageService.java:2861) at org.apache.cassandra.service.StorageService.decommission(StorageService.java:2808) Now I'm in a state where the machine is still up but leaving but I can't seem to get it out of the ring. For example: % nodetool removenode 6ba2c7d4-713e-4c14-8df8-f861fb211b0d Exception in thread main java.lang.UnsupportedOperationException: Node /10.0.0.3 is alive and owns this ID. Use decommission command to remove it from the ring Any ideas? /Janne -- Mike Heffner m...@librato.com Librato, Inc.
Re: Failed decommission
This is what I was seeing code wise as well - but Mike's answer was spot on. Glad you got this straightened out. (And huge thanks to Mike for coming back to post a work-around here and on the ticket). On Sun, Aug 25, 2013 at 11:42 AM, Janne Jalkanen janne.jalka...@ecyrd.comwrote: This would be RP (cluster upgraded from 0.6-0.8-1.0-1.1 ;-). Looks to me like decommission assumes Murmur and 64-bit tokens. /Janne On Aug 25, 2013, at 17:25 , Nate McCall n...@thelastpickle.com wrote: Are you using Murmur3 or the older Random partitioner on this cluster? On Sun, Aug 25, 2013 at 3:06 AM, Janne Jalkanen janne.jalka...@ecyrd.comwrote: This on cass 1.2.8 Ring state before decommission -- Address Load Owns Host ID TokenRack UN 10.0.0.1 38.82 GB 33.3% 21a98502-dc74-4ad0-9689-0880aa110409 1 1a UN 10.0.0.2 33.5 GB33.3% cba6b27a-4982-4f04-854d-cc73155d5f69 56713727820156407428984779325531226110 1b UN 10.0.0.3 37.41 GB 0.0% 6ba2c7d4-713e-4c14-8df8-f861fb211b0d 56713727820156407428984779325531226111 1b UN 10.0.0.4 35.7 GB33.3% bf3d4792-f3e0-4062-afe3-be292bc85ed7 11342745564031281485796955865106245 1c Trying to decommission the node ubuntu@10.0.0.3:~$ nodetool decommission Exception in thread main java.lang.NumberFormatException: For input string: 56713727820156407428984779325531226111 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:444) at java.lang.Long.parseLong(Long.java:483) at org.apache.cassandra.service.StorageService.extractExpireTime(StorageService.java:1660) at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1515) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1234) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:949) at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1116) at org.apache.cassandra.service.StorageService.leaveRing(StorageService.java:2817) at org.apache.cassandra.service.StorageService.unbootstrap(StorageService.java:2861) at org.apache.cassandra.service.StorageService.decommission(StorageService.java:2808) Now I'm in a state where the machine is still up but leaving but I can't seem to get it out of the ring. For example: % nodetool removenode 6ba2c7d4-713e-4c14-8df8-f861fb211b0d Exception in thread main java.lang.UnsupportedOperationException: Node / 10.0.0.3 is alive and owns this ID. Use decommission command to remove it from the ring Any ideas? /Janne
Re: Issue with CQLsh
What version of cassandra are you using? On Sun, Aug 25, 2013 at 8:34 AM, Vivek Mishra mishra.v...@gmail.com wrote: Hi, I have created a column family using Cassandra-cli as: create column family default; and then inserted some record as: set default[1]['type']='bytes'; Then i tried to alter table it via cqlsh as: alter table default alter key type text; // it works alter table default alter column1 type text; // it goes for a toss surprisingly any command after that, simple hangs and i need to reset connection. Any suggestions?
Re: conflict resolution in range scans
See that last part on this page: http://wiki.apache.org/cassandra/ReadRepair This doc is dated, but I'm pretty sure it still works this way. On Sun, Aug 25, 2013 at 3:38 PM, John Sanda john.sa...@gmail.com wrote: How is conflict resolution done with range scans? I understand when reading a single column, the latest timestamp wins. Is the timestamp for each column compared with a range scan such that some columns in the result could come from one replica while other columns come from another? - John
Re: Issue with CQLsh
cassandra 1.2.4 On Mon, Aug 26, 2013 at 2:51 AM, Nate McCall n...@thelastpickle.com wrote: What version of cassandra are you using? On Sun, Aug 25, 2013 at 8:34 AM, Vivek Mishra mishra.v...@gmail.comwrote: Hi, I have created a column family using Cassandra-cli as: create column family default; and then inserted some record as: set default[1]['type']='bytes'; Then i tried to alter table it via cqlsh as: alter table default alter key type text; // it works alter table default alter column1 type text; // it goes for a toss surprisingly any command after that, simple hangs and i need to reset connection. Any suggestions?
Re: OrderPreservingPartitioner in 1.2
From the Jira, One possibility is that getToken of OPP can return hex value if it fails to encode bytes to UTF-8 instead of throwing error. By this system tables seem to be working fine with OPP. This looks like an option to try for me. Thanks! (2013/08/23 20:44), Vara Kumar wrote: For the first exception: OPP was not working in 1.2. It has been fixed but not yet there in latest 1.2.8 version. Jira issue about it: https://issues.apache.org/jira/browse/CASSANDRA-5793 On Fri, Aug 23, 2013 at 12:51 PM, Takenori Sato ts...@cloudian.com mailto:ts...@cloudian.com wrote: Hi, I know it has been depreciated, but OrderPreservingPartitioner still works with 1.2? Just wanted to know how it works, but I got a couple of exceptions as below: ERROR [GossipStage:2] 2013-08-23 07:03:57,171 CassandraDaemon.java (line 175) Exception in thread Thread[GossipStage:2,5,main] java.lang.RuntimeException: The provided key was not UTF8 encoded. at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:233) at org.apache.cassandra.dht.OrderPreservingPartitioner.decorateKey(OrderPreservingPartitioner.java:53) at org.apache.cassandra.db.Table.apply(Table.java:379) at org.apache.cassandra.db.Table.apply(Table.java:353) at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:258) at org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:117) at org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:172) at org.apache.cassandra.db.SystemTable.updatePeerInfo(SystemTable.java:258) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1228) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:935) at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:926) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:884) at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:57) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.nio.charset.MalformedInputException: Input length = 1 at java.nio.charset.CoderResult.throwException(CoderResult.java:260) at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781) at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:167) at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:124) at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:229) ... 16 more The key was 0ab68145 in HEX, that contains some control characters. Another exception is this: INFO [main] 2013-08-23 07:04:27,659 StorageService.java (line 891) JOINING: Starting to bootstrap... DEBUG [main] 2013-08-23 07:04:27,659 BootStrapper.java (line 73) Beginning bootstrap process ERROR [main] 2013-08-23 07:04:27,666 CassandraDaemon.java (line 430) Exception encountered during startup java.lang.IllegalStateException: No sources found for (H,H] at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:163) at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:924) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:693) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:548) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:445) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:325) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:413) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:456) ERROR [StorageServiceShutdownHook] 2013-08-23 07:04:27,672 CassandraDaemon.java (line 175) Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:362) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:513) at
Re: Issue with CQLsh
My understanding is that if you want to use CQL, you should create your tables via CQL. Mixing thrift calls w/ CQL seems like it's just asking for problems like this. On Sun, Aug 25, 2013 at 6:53 PM, Vivek Mishra mishra.v...@gmail.com wrote: cassandra 1.2.4 On Mon, Aug 26, 2013 at 2:51 AM, Nate McCall n...@thelastpickle.comwrote: What version of cassandra are you using? On Sun, Aug 25, 2013 at 8:34 AM, Vivek Mishra mishra.v...@gmail.comwrote: Hi, I have created a column family using Cassandra-cli as: create column family default; and then inserted some record as: set default[1]['type']='bytes'; Then i tried to alter table it via cqlsh as: alter table default alter key type text; // it works alter table default alter column1 type text; // it goes for a toss surprisingly any command after that, simple hangs and i need to reset connection. Any suggestions? -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: Issue with CQLsh
I understand that CQL - Thrift interoperability is an issue. For Application which were build earlier(using thrift) there must be a way and it should be at least give some error message, but it simply hangs with out any error. -Vivek On Mon, Aug 26, 2013 at 8:42 AM, Jonathan Haddad j...@jonhaddad.com wrote: My understanding is that if you want to use CQL, you should create your tables via CQL. Mixing thrift calls w/ CQL seems like it's just asking for problems like this. On Sun, Aug 25, 2013 at 6:53 PM, Vivek Mishra mishra.v...@gmail.comwrote: cassandra 1.2.4 On Mon, Aug 26, 2013 at 2:51 AM, Nate McCall n...@thelastpickle.comwrote: What version of cassandra are you using? On Sun, Aug 25, 2013 at 8:34 AM, Vivek Mishra mishra.v...@gmail.comwrote: Hi, I have created a column family using Cassandra-cli as: create column family default; and then inserted some record as: set default[1]['type']='bytes'; Then i tried to alter table it via cqlsh as: alter table default alter key type text; // it works alter table default alter column1 type text; // it goes for a toss surprisingly any command after that, simple hangs and i need to reset connection. Any suggestions? -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Cluster Management
Hi Cassandra Users, Before I go ahead and create my own solution... are there any tools that exist to help with the management of a Cassandra cluster? For example, if I want to make some changes to the configuration file that resides on each node, is there a tool that will propagate the change to each node? Another example is if I want to have a rolling repair (nodetool repair -pr) and clean up running on my cluster, is there a tool that will help manage/configure that? Any feedback would be greatly appreciated. Thanks, Anthony
