Hello,
While testing token allocation with version 3.0.15 we are experiencing some
quite unexpected result.
We have deployed a secondary virtual DC with 6 nodes, 4 tokens per node.
Then we were adding the 7th node to the new DC in order to observe the
effect of ownership re-distribution.
To set up the new DC we've used the following steps:
1. Alter all keyspaces to replicate to the upcoming new DC.
2. Deploy 3 seed nodes (IP ends with .31) with num_tokens=4 and tokens
specified by initial_token list, auto_bootstrap=false.
3. Deploy 3 more nodes (IP ends with .32) with num_tokens=4 and
allocate_tokens_for_keyspace=data_ks, auto_bootstrap=true.
4. Rebuild all new nodes specifying eu-central as the source DC (for the 3
already bootstrapped nodes, workaround by truncating
system.available_ranges first).
The following is the output of nodetool status after starting to bootstrap
the 7th node (172.31.128.33):
Datacenter: eu-central
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 172.31.160.12 26.4 GB 256 48.9%
89067222-b0eb-49e5-be7d-758ea24ace9a 1c
UN 172.31.144.12 28.92 GB 256 52.6%
2ab4786f-9722-4418-ba78-9c435cbb30e5 1b
UN 172.31.128.12 28.13 GB 256 47.9%
c4733a5c-abc5-4bab-9449-1e3f584cf64f 1a
UN 172.31.128.11 29.84 GB 256 52.2%
6083369c-1a0f-4098-a420-313dacd429b6 1a
UN 172.31.160.11 28.25 GB 256 51.1%
4dc361fc-818a-4b7f-abd3-9121488a7db1 1c
UN 172.31.144.11 28.14 GB 256 47.4%
05e5df92-d196-46d5-8812-e843fbbd2922 1b
Datacenter: eu-central_4vn
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 172.31.128.31 24.83 GB 4 45.8%
4d7decb3-8692-4aec-a2e1-2ac89aed8c5a 1a
UN 172.31.144.31 26.52 GB 4 45.8%
2eb29602-2df5-4f4f-b419-b5a94cf785f0 1b
UN 172.31.160.31 24.8 GB 4 45.8%
f1bd4696-c25c-4bc3-8c30-292f2bd027c1 1c
UJ 172.31.128.33 568.94 MB 4 ?
ffa21d50-9bb4-4d2b-9e3e-7a6945f6f071 1a
UN 172.31.144.32 29.3 GB 4 54.2%
5ce019f6-99fd-4333-b231-d04a266229bb 1b
UN 172.31.160.32 27.8 GB 4 54.2%
193bef27-eea8-4aa6-9d5f-8baf3decdd76 1c
UN 172.31.128.32 30.5 GB 4 54.2%
6a046b64-31f9-4881-85b0-ab3a2f6dcdc4 1a
Then we wanted to start testing distribution with 8 vnodes. For that we
started to deploy yet another DC.
The following is the output of nodetool status after deploying the 3 seed
nodes of the 8-tokens DC:
Datacenter: eu-central
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 172.31.160.12 26.4 GB 256 48.9%
89067222-b0eb-49e5-be7d-758ea24ace9a 1c
UN 172.31.144.12 28.92 GB 256 52.6%
2ab4786f-9722-4418-ba78-9c435cbb30e5 1b
UN 172.31.128.12 28.13 GB 256 47.9%
c4733a5c-abc5-4bab-9449-1e3f584cf64f 1a
UN 172.31.128.11 29.84 GB 256 52.2%
6083369c-1a0f-4098-a420-313dacd429b6 1a
UN 172.31.160.11 28.25 GB 256 51.1%
4dc361fc-818a-4b7f-abd3-9121488a7db1 1c
UN 172.31.144.11 28.14 GB 256 47.4%
05e5df92-d196-46d5-8812-e843fbbd2922 1b
Datacenter: eu-central_4vn
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
*UN 172.31.128.31 24.83 GB 3 45.8%
4d7decb3-8692-4aec-a2e1-2ac89aed8c5a 1a*
UN 172.31.144.31 26.52 GB 4 45.8%
2eb29602-2df5-4f4f-b419-b5a94cf785f0 1b
UN 172.31.160.31 24.8 GB 4 45.8%
f1bd4696-c25c-4bc3-8c30-292f2bd027c1 1c
UJ 172.31.128.33 4.21 GB 4 ?
ffa21d50-9bb4-4d2b-9e3e-7a6945f6f071 1a
UN 172.31.160.32 27.8 GB 4 54.2%
193bef27-eea8-4aa6-9d5f-8baf3decdd76 1c
*UN 172.31.144.32 29.3 GB 3 54.2%
5ce019f6-99fd-4333-b231-d04a266229bb 1b*
UN 172.31.128.32 30.5 GB 4 54.2%
6a046b64-31f9-4881-85b0-ab3a2f6dcdc4 1a
Datacenter: eu-central_8vn
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 172.31.128.41 111.11 KB 8 0.0%
e218b68e-9837-4e6a-acbe-9833fda285bc 1a
UN 172.31.144.41 113.2 KB 8 0.0%
3ec883e9-6b84-4314-85bd-a3c00c4f47c8 1b
UN 172.31.160.41 82.22 KB 8 0.0%
cfaee6c5-ee9c-4d29-aa54-ca3e8e74e356 1c
What is absolutely unexpected is that here we see that 2 nodes in the _4vn
DC apparently now have reduced number of tokens: 3 instead of 4.
How could that happen?
--
Alex
