All DCs in a cluster use the same token space in the DHT, so token conflicts across datacenters are invalid config
-- Jeff Jirsa > On Jan 29, 2018, at 11:50 PM, Oleksandr Shulgin > <oleksandr.shul...@zalando.de> wrote: > >> On Tue, Jan 30, 2018 at 5:13 AM, kurt greaves <k...@instaclustr.com> wrote: >> Shouldn't happen. Can you send through nodetool ring output from one of >> those nodes? Also, did the logs have anything to say about tokens when you >> started the 3 seed nodes? > > Hi Kurt, > > I cannot run nodetool ring anymore, since these test nodes are long gone. > However I've grepped the logs and this is what I've found: > > Jan 25 08:57:18 ip-172-31-128-41 docker/cf3ea463915a[854]: INFO 08:57:18 > Nodes /172.31.128.31 and /172.31.128.41 have the same token > -9223372036854775808. Ignoring /172.31.128.31 > Jan 25 08:57:18 ip-172-31-128-41 docker/cf3ea463915a[854]: INFO 08:57:18 > Nodes /172.31.144.32 and /172.31.128.41 have the same token > -8454757700450211158. Ignoring /172.31.144.32 > Jan 25 08:58:30 ip-172-31-144-41 docker/48fba443d99f[852]: INFO 08:58:30 > Nodes /172.31.128.41 and /172.31.128.31 have the same token > -9223372036854775808. /172.31.128.41 is the new owner > Jan 25 08:58:30 ip-172-31-144-41 docker/48fba443d99f[852]: INFO 08:58:30 > Nodes /172.31.144.32 and /172.31.128.41 have the same token > -8454757700450211158. Ignoring /172.31.144.32 > Jan 25 08:59:45 ip-172-31-160-41 docker/cced70e132f2[849]: INFO 08:59:45 > Nodes /172.31.128.41 and /172.31.128.31 have the same token > -9223372036854775808. /172.31.128.41 is the new owner > Jan 25 08:59:45 ip-172-31-160-41 docker/cced70e132f2[849]: INFO 08:59:45 > Nodes /172.31.144.32 and /172.31.128.41 have the same token > -8454757700450211158. Ignoring /172.31.144.32 > > Since we are allocating the tokens for seed nodes manually, it appears that > the first seed node in the new ring (172.31.128.41) gets the same first token > (-9223372036854775808) as the node in the old ring (172.31.128.31). The same > goes for the 3rd token of the new seed node (-8454757700450211158). > > What is beyond me is why would that matter and why would token ownership > change at all, while these nodes are in the *different virtual DCs*? To me > this sounds like a paticularly nasty bug... > > -- > Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 > 127-59-707 >
