[
https://issues.apache.org/jira/browse/CASSANDRA-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-5916:
----------------------------------------
Attachment: 5916-v3.txt
There are two distinct cases here: we replace 'ourself' with the same IP, or we
replace a dead node with a new IP (ala ec2.) We can't know which one we're
doing _a priori_, so we shadow gossip. If we're replacing the same IP, our
shadow SYN will contain it, and the remote node will ACK with what we need.
If we're _not_ replacing with the same IP, there's a problem: an ACK will only
contain what was present in the SYN digest list. One could argue this is the
sender being naive, since it obviously knows the node that sent the SYN doesn't
have some states that it does, but I think at scale this makes sense since it's
possible a third node has begun gossiping with the SYN sender, too. In any
case, I don't want to change that behavior at this point.
The other problem is, we can't just sit around and wait for someone to send us
a populated SYN either, since we're not a part of gossip and we're new. But we
don't know we're new yet, and can't insert ourselves into gossip either, or
we'll break the case of using the same IP.
So, we'll create a special case for shadow gossip, and redefine it a bit.
Instead of sending a SYN with our own endpoint and a generation of zero, we'll
send a completely empty SYN (digest-wise, we still populate the cluster name
and partioner, since those checks still make sense.) This won't ever normally
occur in gossip, because a node always knows about and adds itself. When we
see an empty SYN, we can know that the node that sent it is asking for
everything we've got, and we can ACK with just that, allowing the replacement
node to have whatever it needs for either the same or different IP cases.
v3 does this.
> gossip and tokenMetadata get hostId out of sync on failed replace_node with
> the same IP address
> -----------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-5916
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5916
> Project: Cassandra
> Issue Type: Bug
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.2.12
>
> Attachments: 5916.txt, 5916-v2.txt, 5916-v3.txt
>
>
> If you try to replace_node an existing, live hostId, it will error out.
> However if you're using an existing IP to do this (as in, you chose the wrong
> uuid to replace on accident) then the newly generated hostId wipes out the
> old one in TMD, and when you do try to replace it replace_node will complain
> it does not exist. Examination of gossipinfo still shows the old hostId,
> however now you can't replace it either.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
