Re: Drop in node replacements.
On Sat, Apr 5, 2014 at 5:10 PM, Anand Somani meatfor...@gmail.com wrote: Have you tried nodetool rebuild for that node? I have seen that work when repair failed. While rebuild may work in cases when repair doesn't, they do different things and are not mutually substitutable. rebuild is essentially bootstrap, streaming data from one source replica per range. repair syncs all replicas per range. If you can successful rebuild, it is true that afterwards your delta to repair will be much smaller and potentially more likely to actually succeed. =Rob
Re: Drop in node replacements.
Have you tried nodetool rebuild for that node? I have seen that work when repair failed. On Wed, Apr 2, 2014 at 11:44 AM, Redmumba redmu...@gmail.com wrote: Cassandra 1.2.15, using commodity hardware. On Tue, Apr 1, 2014 at 6:37 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Apr 1, 2014 at 3:24 PM, Redmumba redmu...@gmail.com wrote: Is it possible to have true drop in node replacements? For example, I have a cluster of 51 Cassandra nodes, 17 in each data center. I had one host go down on DC3, and when it came back up, it joined the ring, etc., but was not receiving any data. Even after multiple restarts and forcing a repair on the entire fleet, it still holds maybe ~30MB on a cluster that is absorbing ~1.2TB a day. What version of Cassandra? Real hardware/network or virtual? =Rob
Re: Drop in node replacements.
Cassandra 1.2.15, using commodity hardware. On Tue, Apr 1, 2014 at 6:37 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Apr 1, 2014 at 3:24 PM, Redmumba redmu...@gmail.com wrote: Is it possible to have true drop in node replacements? For example, I have a cluster of 51 Cassandra nodes, 17 in each data center. I had one host go down on DC3, and when it came back up, it joined the ring, etc., but was not receiving any data. Even after multiple restarts and forcing a repair on the entire fleet, it still holds maybe ~30MB on a cluster that is absorbing ~1.2TB a day. What version of Cassandra? Real hardware/network or virtual? =Rob
Drop in node replacements.
Is it possible to have true drop in node replacements? For example, I have a cluster of 51 Cassandra nodes, 17 in each data center. I had one host go down on DC3, and when it came back up, it joined the ring, etc., but was not receiving any data. Even after multiple restarts and forcing a repair on the entire fleet, it still holds maybe ~30MB on a cluster that is absorbing ~1.2TB a day. On top of that, I decided to see if I could recreate it--by taking down a node, reprovisioning it, and then throwing it back in WITHOUT having it take over the old node's tokens, it never seems to ever absorb any of the old data after a full repair, and it never seems to start loading new data (I now have 3 nodes using ~30MB). Am I doing something wrong? I would imagine a repair on the entire cluster (across 3 DCs) would force C* to put some copies onto the other node--but this doesn't seem to be the case. What can I do? Andrew
Re: Drop in node replacements.
On Tue, Apr 1, 2014 at 3:24 PM, Redmumba redmu...@gmail.com wrote: Is it possible to have true drop in node replacements? For example, I have a cluster of 51 Cassandra nodes, 17 in each data center. I had one host go down on DC3, and when it came back up, it joined the ring, etc., but was not receiving any data. Even after multiple restarts and forcing a repair on the entire fleet, it still holds maybe ~30MB on a cluster that is absorbing ~1.2TB a day. What version of Cassandra? Real hardware/network or virtual? =Rob
