Thanks Aaron. I shall investigate more next week about this.
regards. Jean Armel 2013/2/22 aaron morton <[email protected]> > So, it looks that the repair is required if we want to add new nodes in > our platform, but I don't understand why. > > Bootstrapping should take care of it. But new seed nodes do not bootstrap. > Check the logs on the nodes you added to see what messages have "bootstrap" > in them. > > Anytime you are worried about things like this throw in a nodetool repair. > If you are using QUOURM for read and writes you will still be getting > consistent data, so long as you have only added one node. Or one node every > RF'th nodes. > > Cheers > > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 22/02/2013, at 9:55 PM, Jean-Armel Luce <[email protected]> wrote: > > Hi Aaron, > > Thanks for your answer. > > > I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes > instead of 16 (it is a test cluster). > There are 2 datacenters b1 and s1. > > Here is the result of nodetool status after adding a new node in the 1st > datacenter (dc s1): > root@node007:~# nodetool status > Datacenter: b1 > ============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host > ID Rack > UN 10.234.72.135 10.71 GB 256 44.6% > 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01 > UN 10.234.72.134 16.74 GB 256 63.7% > f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01 > UN 10.234.72.139 17.09 GB 256 62.0% > 95661392-ccd8-4592-a76f-1c99f7cdf23a e07 > UN 10.234.72.138 10.96 GB 256 42.9% > 0d6725f0-1357-423d-85c1-153fb94257d5 e03 > UN 10.234.72.137 11.09 GB 256 45.7% > 492190d7-3055-4167-8699-9c6560e28164 e03 > UN 10.234.72.136 11.91 GB 256 41.1% > 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01 > Datacenter: s1 > ============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host > ID Rack > UN 10.98.255.139 16.94 GB 256 43.8% > 3523e80c-8468-4502-b334-79eabc3357f0 g10 > UN 10.98.255.138 12.62 GB 256 42.4% > a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02 > UN 10.98.255.137 10.59 GB 256 38.4% > f851b6ee-f1e4-431b-8beb-e7b173a77342 i02 > UN 10.98.255.136 11.89 GB 256 42.9% > 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09 > UN 10.98.255.135 10.29 GB 256 40.4% > e2d020a5-97a9-48d4-870c-d10b59858763 a09 > UN 10.98.255.134 16.19 GB 256 52.3% > 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06 > UN 10.98.255.140 127.84 KB 256 39.9% > 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10 > > We can see that the new node (10.98.255.140) contains only 127,84KB. > We saw also that there was no network traffic between the nodes. > > Then we added a new node in the 2nd datacenter (dc b1) > > > > root@node007:~# nodetool status > Datacenter: b1 > ============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host > ID Rack > UN 10.234.72.135 12.95 GB 256 42.0% > 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01 > UN 10.234.72.134 20.11 GB 256 53.1% > f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01 > UN 10.234.72.140 122.25 KB 256 41.9% > 501ea498-8fed-4cc8-a23a-c99492bc4f26 e07 > UN 10.234.72.139 20.46 GB 256 40.2% > 95661392-ccd8-4592-a76f-1c99f7cdf23a e07 > UN 10.234.72.138 13.21 GB 256 40.9% > 0d6725f0-1357-423d-85c1-153fb94257d5 e03 > UN 10.234.72.137 13.34 GB 256 42.9% > 492190d7-3055-4167-8699-9c6560e28164 e03 > UN 10.234.72.136 14.16 GB 256 39.0% > 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01 > Datacenter: s1 > ============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host > ID Rack > UN 10.98.255.139 19.19 GB 256 43.8% > 3523e80c-8468-4502-b334-79eabc3357f0 g10 > UN 10.98.255.138 14.9 GB 256 42.4% > a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02 > UN 10.98.255.137 12.49 GB 256 38.4% > f851b6ee-f1e4-431b-8beb-e7b173a77342 i02 > UN 10.98.255.136 14.13 GB 256 42.9% > 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09 > UN 10.98.255.135 12.16 GB 256 40.4% > e2d020a5-97a9-48d4-870c-d10b59858763 a09 > UN 10.98.255.134 18.85 GB 256 52.3% > 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06 > UN 10.98.255.140 2.24 GB 256 39.9% > 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10 > > > We can see that the 2nd new node (10.234.72.140) contains only 122,25KB. > The new node in the 1st datacenter contains now 2,24 GB because we were > inserting data in the cluster while adding the new nodes. > > Then we started a repair from the new node in the 2nd datacenter : > time nodetool repair > > > We can see that the old nodes are sending data to the new node : > root@node007:~# nodetool netstats > Mode: NORMAL > Not sending any streams. > Streaming from: /10.98.255.137 > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db > sections=130 progress=0/15598366 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db > sections=107 progress=0/429517 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db > sections=109 progress=0/696057 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db > sections=57 progress=0/189844 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db > sections=124 progress=56492032/4597955 - 1228% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db > sections=113 progress=0/850632 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db > sections=130 progress=0/16815423 - 0% > Streaming from: /10.98.255.136 > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db > sections=104 progress=0/3545520 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db > sections=43 progress=0/252976 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db > sections=91 progress=0/443365 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db > sections=101 progress=11337728/4188624 - 270% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db > sections=94 progress=0/758418 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db > sections=88 progress=0/396552 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db > sections=105 progress=0/12363907 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db > sections=87 progress=0/506537 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db > sections=42 progress=0/197531 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db > sections=110 progress=0/12689898 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db > sections=83 progress=0/379628 - 0% > Streaming from: /10.98.255.134 > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db > sections=108 progress=0/10715348 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db > sections=82 progress=0/8177161 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db > sections=84 progress=25613959168/8556942 - 299335% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db > sections=79 progress=0/506293 - 0% > Pool Name Active Pending Completed > Commands n/a 0 835 > Responses n/a 0 6765 > > So, it looks that the repair is required if we want to add new nodes in > our platform, but I don't understand why. > > Regards. > > Jean Armel > > > >
