What's the partitioner you use? We have logic to prevent duplicate tokens.
private static Collection<Token> adjustForCrossDatacenterClashes(final
TokenMetadata tokenMetadata,
StrategyAdapter strategy, Collection<Token> tokens)
{
List<Token> filtered = Lists.newArrayListWithCapacity(tokens.size());
for (Token t : tokens)
{
while (tokenMetadata.getEndpoint(t) != null)
{
InetAddress other = tokenMetadata.getEndpoint(t);
if (strategy.inAllocationRing(other))
throw new
ConfigurationException(String.format("Allocated token %s already
assigned to node %s. Is another node also allocating tokens?", t,
other));
t = t.increaseSlightly();
}
filtered.add(t);
}
return filtered;
}
On Tue, Jan 30, 2018 at 8:44 AM, Jeff Jirsa <[email protected]> wrote:
> All DCs in a cluster use the same token space in the DHT, so token
> conflicts across datacenters are invalid config
>
>
> --
> Jeff Jirsa
>
>
> On Jan 29, 2018, at 11:50 PM, Oleksandr Shulgin <
> [email protected]> wrote:
>
> On Tue, Jan 30, 2018 at 5:13 AM, kurt greaves <[email protected]>
> wrote:
>
>> Shouldn't happen. Can you send through nodetool ring output from one of
>> those nodes? Also, did the logs have anything to say about tokens when you
>> started the 3 seed nodes?
>>
>
> Hi Kurt,
>
> I cannot run nodetool ring anymore, since these test nodes are long gone.
> However I've grepped the logs and this is what I've found:
>
> Jan 25 08:57:18 ip-172-31-128-41 docker/cf3ea463915a[854]: INFO 08:57:18
> Nodes /172.31.128.31 and /172.31.128.41 have the same token
> -9223372036854775808. Ignoring /172.31.128.31
> Jan 25 08:57:18 ip-172-31-128-41 docker/cf3ea463915a[854]: INFO 08:57:18
> Nodes /172.31.144.32 and /172.31.128.41 have the same token
> -8454757700450211158. Ignoring /172.31.144.32
> Jan 25 08:58:30 ip-172-31-144-41 docker/48fba443d99f[852]: INFO 08:58:30
> Nodes /172.31.128.41 and /172.31.128.31 have the same token
> -9223372036854775808. /172.31.128.41 is the new owner
> Jan 25 08:58:30 ip-172-31-144-41 docker/48fba443d99f[852]: INFO 08:58:30
> Nodes /172.31.144.32 and /172.31.128.41 have the same token
> -8454757700450211158. Ignoring /172.31.144.32
> Jan 25 08:59:45 ip-172-31-160-41 docker/cced70e132f2[849]: INFO 08:59:45
> Nodes /172.31.128.41 and /172.31.128.31 have the same token
> -9223372036854775808. /172.31.128.41 is the new owner
> Jan 25 08:59:45 ip-172-31-160-41 docker/cced70e132f2[849]: INFO 08:59:45
> Nodes /172.31.144.32 and /172.31.128.41 have the same token
> -8454757700450211158. Ignoring /172.31.144.32
>
> Since we are allocating the tokens for seed nodes manually, it appears
> that the first seed node in the new ring (172.31.128.41) gets the same
> first token (-9223372036854775808) as the node in the old ring
> (172.31.128.31). The same goes for the 3rd token of the new seed node
> (-8454757700450211158).
>
> What is beyond me is why would that matter and why would token ownership
> change at all, while these nodes are in the *different virtual DCs*? To me
> this sounds like a paticularly nasty bug...
>
> --
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707 <+49%20176%2012759707>
>
>
--
Dikang
