Decommissioning a datacenter deletes the data (on decommissioned datacenter)

2014-08-07 Thread srmore
tl;dr: Decommissioning datacenters by running nodetool decommission on a node deletes the data on the decommissioned node - is this expected ? I am trying our some tests on my multi-datacenter setup. Somewhere in the docs I read that decommissioning a node will stream its data to other nodes but

Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)

2014-08-07 Thread Robert Coli
On Thu, Aug 7, 2014 at 8:26 AM, srmore comom...@gmail.com wrote: tl;dr: Decommissioning datacenters by running nodetool decommission on a node deletes the data on the decommissioned node - is this expected ? What does deletes mean? What does lost all my keyspaces (and data) mean? =Rob

Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)

2014-08-07 Thread srmore
Hello Rob Sorry for being ambiguous. By deletes I mean that running decommission I can no longer see any keyspaces owned by this node or replicated by other nodes using the cfstats command. I am also seeing the same behavior when I remove a single node from a cluster (without datacenters). On

Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)

2014-08-07 Thread Robert Coli
On Thu, Aug 7, 2014 at 10:04 AM, srmore comom...@gmail.com wrote: Sorry for being ambiguous. By deletes I mean that running decommission I can no longer see any keyspaces owned by this node or replicated by other nodes using the cfstats command. I am also seeing the same behavior when I

Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)

2014-08-07 Thread srmore
On Thu, Aug 7, 2014 at 12:27 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 7, 2014 at 10:04 AM, srmore comom...@gmail.com wrote: Sorry for being ambiguous. By deletes I mean that running decommission I can no longer see any keyspaces owned by this node or replicated by other nodes

Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)

2014-08-07 Thread Ken Hancock
My reading is it didn't forget the schema. It lost the data. My reading is decomissioning worked fine. Possibly when you changed the replication on a keyspace to include a second data center, the data didn't get replicated. When you ADD a datacenter, you need to do a nodetool rebuild to get

Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)

2014-08-07 Thread srmore
Thanks for the detailed reply Ken, this really helps. I also realized that I wasn't doing a 'nodetool rebuild' after reading your email. I was following the steps mentioned here http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_decomission_dc_t.html I do a test with

Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)

2014-08-07 Thread srmore
I tried using 'nodetool rebuild' after I add the datacenters,date same outcome, and after I decommission my keyspaces are getting wiped out, I don't understand this. On Thu, Aug 7, 2014 at 1:54 PM, srmore comom...@gmail.com wrote: Thanks for the detailed reply Ken, this really helps. I also

Re: Decommissioning a datacenter deletes the data (on decommissioned datacenter)

2014-08-07 Thread Robert Coli
On Thu, Aug 7, 2014 at 12:29 PM, srmore comom...@gmail.com wrote: I tried using 'nodetool rebuild' after I add the datacenters,date same outcome, and after I decommission my keyspaces are getting wiped out, I don't understand this. File a JIRA with steps to reproduce the issue, and someone