Re: Repair doesn't work after upgrading to 0.8.1
Hi All, sorry for taking so long to answer. I was away from the internet. Héctor, when you say I have upgraded all my cluster to 0.8.1, from which version was that: 0.7.something or 0.8.0 ? 0.7.6-2 to 0.8.1 This is the same behavior I reported in 2768 as Aaron referenced ... What was suggested for us was to do the following: - Shut down the entire ring - When you bring up each node, do a nodetool repair That's exactly what I ended up doing. Repair now works. I tried to do a rolling restart with 2818 applied, but it did not work. However, in the issue reported, it was unable to be reproduced ... I'd be curious to know how Hector's keyspace is defined. Ours at the time was RF=3 and using Ec2 snitch... Nothing special, Default snithch, RF=3. I think this should be prioritized, as having to restart the whole cluster is a bit extreme. We don't have separate DCs, so I had to incurre on downtime, which costs money, and a little bit of grief. El vie, 01-07-2011 a las 10:16 +0200, Sylvain Lebresne escribió: To make it clear what the problem is, this is not a repair problem. This is a gossip problem. Gossip is reporting that the remote node is a 0.7 node and repair is just saying I cannot use that node because repair has changed and the 0.7 node will not know how to answer me correctly, which is the correct behavior if the node happens to be a 0.7 node. Hence, I'm kind of baffled that dropping a keyspace and recreating it fixed anything. Unless as part of removed the keyspace, you've deleted the system tables, in which case that could have triggered something. -- Sylvain On Fri, Jul 1, 2011 at 9:33 AM, Sasha Dolgy sdo...@gmail.com wrote: This is the same behavior I reported in 2768 as Aaron referenced ... What was suggested for us was to do the following: - Shut down the entire ring - When you bring up each node, do a nodetool repair That didn't immediately resolve the problems. In the end, I backed up all the data, removed the keyspace and created a new one. That seemed to have solved our problems. That was from 0.7.6-2 to 0.8.0 However, in the issue reported, it was unable to be reproduced ... I'd be curious to know how Hector's keyspace is defined. Ours at the time was RF=3 and using Ec2 snitch... -sd On Fri, Jul 1, 2011 at 9:22 AM, Sylvain Lebresne sylv...@datastax.com wrote: Héctor, when you say I have upgraded all my cluster to 0.8.1, from which version was that: 0.7.something or 0.8.0 ? If this was 0.8.0, did you run successful repair on 0.8.0 previous to the upgrade ?
Re: Repair doesn't work after upgrading to 0.8.1
On Fri, Jul 1, 2011 at 3:16 AM, Sylvain Lebresne sylv...@datastax.com wrote: To make it clear what the problem is, this is not a repair problem. This is a gossip problem. Gossip is reporting that the remote node is a 0.7 node and repair is just saying I cannot use that node because repair has changed and the 0.7 node will not know how to answer me correctly, which is the correct behavior if the node happens to be a 0.7 node. Technically, this is not part of gossip (in that no state is being gossiped for this, but we do maintain this state in the Gossiper class), but your analysis of the problem is correct. The problem is that on an upgrade via rolling restart, the existing nodes still remember the new ones as being old, so they mimic the old version, thusly propagating the old version around. Hence, I'm kind of baffled that dropping a keyspace and recreating it fixed anything. Unless as part of removed the keyspace, you've deleted the system tables, in which case that could have triggered something. I don't see how this could help either, since the version is bound in Gossiper and set by IncomingTcpConnection. I've created https://issues.apache.org/jira/browse/CASSANDRA-2860 to get this resolved. --Brandon
Re: Repair doesn't work after upgrading to 0.8.1
Héctor, when you say I have upgraded all my cluster to 0.8.1, from which version was that: 0.7.something or 0.8.0 ? If this was 0.8.0, did you run successful repair on 0.8.0 previous to the upgrade ? -- Sylvain On Fri, Jul 1, 2011 at 5:59 AM, Terje Marthinussen tmarthinus...@gmail.com wrote: Unless it is a 0.8.1 RC or beta On Fri, Jul 1, 2011 at 12:57 PM, Jonathan Ellis jbel...@gmail.com wrote: This isn't 2818 -- (a) the 0.8.1 protocol is identical to 0.8.0 and (b) the whole cluster is on the same version. On Thu, Jun 30, 2011 at 9:35 PM, aaron morton aa...@thelastpickle.com wrote: This seems to be a known issue related to https://issues.apache.org/jira/browse/CASSANDRA-2818 e.g. https://issues.apache.org/jira/browse/CASSANDRA-2768 There was some discussion on the IRC list today, driftx said the simple fix was a full cluster restart. Or perhaps a rolling restart with the 2818 patch applied may work. Starting with Dcassandra.load_ring_state=false causes the node to rediscover the ring which may help (just a guess really). But if there is bad node start been passed around in gossip it will just get the bad state again. Anyone else ? - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 1 Jul 2011, at 09:11, Héctor Izquierdo Seliva wrote: Hi all, I have upgraded all my cluster to 0.8.1. Today one of the disks in one of the nodes died. After replacing the disk I tried running repair, but this message appears: INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.77 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (170141183460469231731687303715884105727,28356863910078205288614550619314017621]: manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098 completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.79 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (141784319550391026443072753096570088105,170141183460469231731687303715884105727]: manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,087 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (113427455640312821154458202477256070484,141784319550391026443072753096570088105]: manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a completed. What can I do? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Repair doesn't work after upgrading to 0.8.1
This is the same behavior I reported in 2768 as Aaron referenced ... What was suggested for us was to do the following: - Shut down the entire ring - When you bring up each node, do a nodetool repair That didn't immediately resolve the problems. In the end, I backed up all the data, removed the keyspace and created a new one. That seemed to have solved our problems. That was from 0.7.6-2 to 0.8.0 However, in the issue reported, it was unable to be reproduced ... I'd be curious to know how Hector's keyspace is defined. Ours at the time was RF=3 and using Ec2 snitch... -sd On Fri, Jul 1, 2011 at 9:22 AM, Sylvain Lebresne sylv...@datastax.com wrote: Héctor, when you say I have upgraded all my cluster to 0.8.1, from which version was that: 0.7.something or 0.8.0 ? If this was 0.8.0, did you run successful repair on 0.8.0 previous to the upgrade ?
Re: Repair doesn't work after upgrading to 0.8.1
To make it clear what the problem is, this is not a repair problem. This is a gossip problem. Gossip is reporting that the remote node is a 0.7 node and repair is just saying I cannot use that node because repair has changed and the 0.7 node will not know how to answer me correctly, which is the correct behavior if the node happens to be a 0.7 node. Hence, I'm kind of baffled that dropping a keyspace and recreating it fixed anything. Unless as part of removed the keyspace, you've deleted the system tables, in which case that could have triggered something. -- Sylvain On Fri, Jul 1, 2011 at 9:33 AM, Sasha Dolgy sdo...@gmail.com wrote: This is the same behavior I reported in 2768 as Aaron referenced ... What was suggested for us was to do the following: - Shut down the entire ring - When you bring up each node, do a nodetool repair That didn't immediately resolve the problems. In the end, I backed up all the data, removed the keyspace and created a new one. That seemed to have solved our problems. That was from 0.7.6-2 to 0.8.0 However, in the issue reported, it was unable to be reproduced ... I'd be curious to know how Hector's keyspace is defined. Ours at the time was RF=3 and using Ec2 snitch... -sd On Fri, Jul 1, 2011 at 9:22 AM, Sylvain Lebresne sylv...@datastax.com wrote: Héctor, when you say I have upgraded all my cluster to 0.8.1, from which version was that: 0.7.something or 0.8.0 ? If this was 0.8.0, did you run successful repair on 0.8.0 previous to the upgrade ?
Repair doesn't work after upgrading to 0.8.1
Hi all, I have upgraded all my cluster to 0.8.1. Today one of the disks in one of the nodes died. After replacing the disk I tried running repair, but this message appears: INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.77 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (170141183460469231731687303715884105727,28356863910078205288614550619314017621]: manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098 completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.79 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (141784319550391026443072753096570088105,170141183460469231731687303715884105727]: manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,087 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (113427455640312821154458202477256070484,141784319550391026443072753096570088105]: manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a completed. What can I do?
Re: Repair doesn't work after upgrading to 0.8.1
This seems to be a known issue related to https://issues.apache.org/jira/browse/CASSANDRA-2818 e.g. https://issues.apache.org/jira/browse/CASSANDRA-2768 There was some discussion on the IRC list today, driftx said the simple fix was a full cluster restart. Or perhaps a rolling restart with the 2818 patch applied may work. Starting with Dcassandra.load_ring_state=false causes the node to rediscover the ring which may help (just a guess really). But if there is bad node start been passed around in gossip it will just get the bad state again. Anyone else ? - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 1 Jul 2011, at 09:11, Héctor Izquierdo Seliva wrote: Hi all, I have upgraded all my cluster to 0.8.1. Today one of the disks in one of the nodes died. After replacing the disk I tried running repair, but this message appears: INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.77 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (170141183460469231731687303715884105727,28356863910078205288614550619314017621]: manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098 completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.79 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (141784319550391026443072753096570088105,170141183460469231731687303715884105727]: manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,087 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (113427455640312821154458202477256070484,141784319550391026443072753096570088105]: manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a completed. What can I do?
Re: Repair doesn't work after upgrading to 0.8.1
This isn't 2818 -- (a) the 0.8.1 protocol is identical to 0.8.0 and (b) the whole cluster is on the same version. On Thu, Jun 30, 2011 at 9:35 PM, aaron morton aa...@thelastpickle.com wrote: This seems to be a known issue related to https://issues.apache.org/jira/browse/CASSANDRA-2818 e.g. https://issues.apache.org/jira/browse/CASSANDRA-2768 There was some discussion on the IRC list today, driftx said the simple fix was a full cluster restart. Or perhaps a rolling restart with the 2818 patch applied may work. Starting with Dcassandra.load_ring_state=false causes the node to rediscover the ring which may help (just a guess really). But if there is bad node start been passed around in gossip it will just get the bad state again. Anyone else ? - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 1 Jul 2011, at 09:11, Héctor Izquierdo Seliva wrote: Hi all, I have upgraded all my cluster to 0.8.1. Today one of the disks in one of the nodes died. After replacing the disk I tried running repair, but this message appears: INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.77 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (170141183460469231731687303715884105727,28356863910078205288614550619314017621]: manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098 completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.79 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (141784319550391026443072753096570088105,170141183460469231731687303715884105727]: manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,087 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (113427455640312821154458202477256070484,141784319550391026443072753096570088105]: manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a completed. What can I do? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Repair doesn't work after upgrading to 0.8.1
Unless it is a 0.8.1 RC or beta On Fri, Jul 1, 2011 at 12:57 PM, Jonathan Ellis jbel...@gmail.com wrote: This isn't 2818 -- (a) the 0.8.1 protocol is identical to 0.8.0 and (b) the whole cluster is on the same version. On Thu, Jun 30, 2011 at 9:35 PM, aaron morton aa...@thelastpickle.com wrote: This seems to be a known issue related to https://issues.apache.org/jira/browse/CASSANDRA-2818 e.g. https://issues.apache.org/jira/browse/CASSANDRA-2768 There was some discussion on the IRC list today, driftx said the simple fix was a full cluster restart. Or perhaps a rolling restart with the 2818 patch applied may work. Starting with Dcassandra.load_ring_state=false causes the node to rediscover the ring which may help (just a guess really). But if there is bad node start been passed around in gossip it will just get the bad state again. Anyone else ? - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 1 Jul 2011, at 09:11, Héctor Izquierdo Seliva wrote: Hi all, I have upgraded all my cluster to 0.8.1. Today one of the disks in one of the nodes died. After replacing the disk I tried running repair, but this message appears: INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.80 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.77 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,085 AntiEntropyService.java (line 179) Excluding /10.20.13.76 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (170141183460469231731687303715884105727,28356863910078205288614550619314017621]: manual-repair-26f5a7dd-cf12-44de-9f8f-6b6335bdd098 completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 179) Excluding /10.20.13.79 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30 20:36:25,086 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (141784319550391026443072753096570088105,170141183460469231731687303715884105727]: manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf completed. INFO [manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a] 2011-06-30 20:36:25,087 AntiEntropyService.java (line 782) No neighbors to repair with for sbs on (113427455640312821154458202477256070484,141784319550391026443072753096570088105]: manual-repair-2a11d01c-e1e4-4f1e-b8cd-00a9a3fd2f4a completed. What can I do? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com