Re: Cluster Maintenance Mishap

On 21 October 2016 at 15:15, Branton Davis wrote: > For example, I forgot to mention until I read your comment, that the > instances showed as UN (up, normal) instead of UJ (up, joining) while they > were apparently bootstrapping. It's likely these nodes were

Re: Cluster Maintenance Mishap

Thanks. Unfortunately, we lost our system logs during all of this (had normal logs, but not system) due to an unrelated issue :/ Anyhow, as far as I can tell, we're doing okay. On Thu, Oct 20, 2016 at 11:18 PM, Jeremiah D Jordan < jeremiah.jor...@gmail.com> wrote: > The easiest way to figure

Re: Cluster Maintenance Mishap

It mostly seems so. The thing that bugs me is that some things acted like they weren't joining as a normal new node. For example, I forgot to mention until I read your comment, that the instances showed as UN (up, normal) instead of UJ (up, joining) while they were apparently bootstrapping.

Re: Cluster Maintenance Mishap

On 20 October 2016 at 20:58, Branton Davis wrote: > Would they have taken on the token ranges of the original nodes or acted > like new nodes and got new token ranges? If the latter, is it possible > that any data moved from the healthy nodes to the "new" nodes or >

Re: Cluster Maintenance Mishap

The easiest way to figure out what happened is to examine the system log. It will tell you what happened. But I’m pretty sure your nodes got new tokens during that time. If you want to get back the data inserted during the 2 hours you could use sstableloader to send all the data from the

Re: Cluster Maintenance Mishap

I guess I'm either not understanding how that answers the question and/or I've just a done a terrible job at asking it. I'll sleep on it and maybe I'll think of a better way to describe it tomorrow ;) On Thu, Oct 20, 2016 at 8:45 PM, Yabin Meng wrote: > I believe you're

Re: Cluster Maintenance Mishap

I believe you're using VNodes (because token range change doesn't make sense for single-token setup unless you change it explicitly). If you bootstrap a new node with VNodes, I think the way that the token ranges are assigned to the node is random (I'm not 100% sure here, but should be so

Re: Cluster Maintenance Mishap

Thanks for the response, Yabin. However, if there's an answer to my question here, I'm apparently too dense to see it ;) I understand that, since the system keyspace data was not there, it started bootstrapping. What's not clear is if they took over the token ranges of the previous nodes or got

Re: Cluster Maintenance Mishap

Most likely the issue is caused by the fact that when you move the data, you move the system keyspace data away as well. Meanwhile, due to the error of data being copied into a different location than what C* is expecting, when C* starts, it can not find the system metadata info and therefore