Re: Convert single node C* to cluster (rebalancing problem)

[Jeff Jirsa]

On 2017-06-16 10:31 (-0700), John Hughes  wrote: 
> Hi Affan,
> 
> Others can likely speak to this more authoritatively I am sure, but with a
> RF of 1x, I would not expect it to rebalance. Now if you were 4 nodes and a
> RF of 2x I would expect it to.
> 

Even with an RF of 1, any token range that moves to the new (joining) node will 
result in data transfer.

What won't happen automatically, however, is data being removed from the 
source. "nodetool cleanup" is provided to do that.  Until you run "cleanup", no 
data will be removed from the original hosts.

Note, however, that if you're not sure if things are properly 
bootstrapped/repaired, don't run cleanup until you're happy. If you had RF=1 
and you joined a node with auto_bootstrap=false, the node would immediately 
join the ring and be responsible/own data that it never received, and with RF=1 
there's no way to run repair. You can either remove that node (nodetool 
decommission) and re-add it with auto_bootstrap=true to recover most of the old 
data (this isn't PERFECTLY safe, but it's better than not having the data at 
all).  Alternatively, you could use 'sstableloader' to re-stream the data into 
your cluster.

Again, that's only an issue if you had RF=1 and joined a node with 
auto_bootstrap=false.




-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Convert single node C* to cluster (rebalancing problem)

[John Hughes]

N  XX.XX.XX.XX  17.17 GiB  256  51.3%
>>>>> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
>>>>> UN  XX.XX.XX.XX  26.15 GiB  256  52.4%
>>>>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
>>>>> UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
>>>>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
>>>>> UN  XX.XX.XX.XX  24.39 GiB  256  49.8%
>>>>> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
>>>>> UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
>>>>> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>>>>>
>>>>> Though maybe part of what you are experiencing can be cleared up by
>>>>> repair/compaction/cleanup. Also, what are your outputs when you call out
>>>>> specific keyspaces? Do the numbers get more even?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> auto_bootstrap is true by default. Ensure its set to true. On startup
>>>>>> look at your logs for your auto_bootstrap value.  Look at the node
>>>>>> configuration line in your log file.
>>>>>>
>>>>>> Akhil
>>>>>>
>>>>>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote:
>>>>>>
>>>>>>> No, I didn't set it (left it at default value)
>>>>>>>
>>>>>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Did you make sure auto_bootstrap property is indeed set to [true]
>>>>>>>> when you added the node?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>>>>>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>>>>>>> *To:* Akhil Mehra <akhilme...@gmail.com>
>>>>>>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>;
>>>>>>>> user@cassandra.apache.org
>>>>>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing
>>>>>>>> problem)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> not evenly, i have setup a new cluster with subset of data (around
>>>>>>>> 5gb). using the configuration above I am getting these results
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Datacenter: datacenter1
>>>>>>>>
>>>>>>>> ===
>>>>>>>>
>>>>>>>> Status=Up/Down
>>>>>>>>
>>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>>>
>>>>>>>> --  Address  Load   Tokens   Owns (effective)  Host ID 
>>>>>>>> Rack
>>>>>>>>
>>>>>>>> UN  10.128.2.1   4.86 GiB   256  44.9% 
>>>>>>>> e4427611-c247-42ee-9404-371e177f5f17  rack1
>>>>>>>>
>>>>>>>> UN  10.128.2.10  725.03 MiB  256 55.1% 
>>>>>>>> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>>>>>>>
>>>>>>>> is there anything else I can tweak/check to make the distribution
>>>>>>>> even?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> So now the data is evenly balanced in both nodes?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Refer to the following documentation to get a better understanding
>>>>>>>> of the roc_address and the broadcast_rpc_address
>>>>>>>> https://www.instaclustr.com/demystifying-cassandras-broadcast_address/
>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Fad

Re: Convert single node C* to cluster (rebalancing problem)

[Akhil Mehra]

;
>>>>> On Tue, Jun 13, 2017 at 3:24 AM, Akhil Mehra <akhilme...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Great point John.
>>>>>>
>>>>>> The OP should also note that data distribution also depends on your
>>>>>> schema and incoming data profile.
>>>>>>
>>>>>> If your schema is not modelled correctly you can easily end up
>>>>>> unevenly distributed data.
>>>>>>
>>>>>> Cheers,
>>>>>> Akhil
>>>>>>
>>>>>> On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthug...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Is the OP expecting a perfect 50%/50% split? That, to my experience,
>>>>>>> is not going to happen, it is almost always shifted from a fraction of a
>>>>>>> percent to a couple percent.
>>>>>>>
>>>>>>> Datacenter: eu-west
>>>>>>> ===
>>>>>>> Status=Up/Down
>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>> --  AddressLoad   Tokens   Owns (effective)  Host ID
>>>>>>>   Rack
>>>>>>> UN  XX.XX.XX.XX22.71 GiB  256  47.6%
>>>>>>> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
>>>>>>> UN  XX.XX.XX.XX  17.17 GiB  256  51.3%
>>>>>>> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
>>>>>>> UN  XX.XX.XX.XX  26.15 GiB  256      52.4%
>>>>>>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
>>>>>>> UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
>>>>>>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
>>>>>>> UN  XX.XX.XX.XX  24.39 GiB  256  49.8%
>>>>>>> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
>>>>>>> UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
>>>>>>> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>>>>>>>
>>>>>>> Though maybe part of what you are experiencing can be cleared up by
>>>>>>> repair/compaction/cleanup. Also, what are your outputs when you call out
>>>>>>> specific keyspaces? Do the numbers get more even?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> auto_bootstrap is true by default. Ensure its set to true. On
>>>>>>>> startup look at your logs for your auto_bootstrap value.  Look at the 
>>>>>>>> node
>>>>>>>> configuration line in your log file.
>>>>>>>>
>>>>>>>> Akhil
>>>>>>>>
>>>>>>>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> No, I didn't set it (left it at default value)
>>>>>>>>>
>>>>>>>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Did you make sure auto_bootstrap property is indeed set to
>>>>>>>>>> [true] when you added the node?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>>>>>>>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>>>>>>>>> *To:* Akhil Mehra <akhilme...@gmail.com>
>>>>>>>>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>;
>>>>>>>>>> user@cassandra.apache.org
>>>>>>>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing
>>>>>>>>>> problem)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> not evenly, i have setup a new cluster with subset of data
>>>>>>>>>> (around 5gb). using the configuration above I am getting these 
>&

Re: Convert single node C* to cluster (rebalancing problem)

[Varun Gupta]

oining/Moving
>>>>>> --  AddressLoad   Tokens   Owns (effective)  Host ID
>>>>>>   Rack
>>>>>> UN  XX.XX.XX.XX22.71 GiB  256  47.6%
>>>>>> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
>>>>>> UN  XX.XX.XX.XX  17.17 GiB  256  51.3%
>>>>>> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
>>>>>> UN  XX.XX.XX.XX  26.15 GiB  256  52.4%
>>>>>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
>>>>>> UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
>>>>>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
>>>>>> UN  XX.XX.XX.XX  24.39 GiB  256  49.8%
>>>>>> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
>>>>>> UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
>>>>>> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>>>>>>
>>>>>> Though maybe part of what you are experiencing can be cleared up by
>>>>>> repair/compaction/cleanup. Also, what are your outputs when you call out
>>>>>> specific keyspaces? Do the numbers get more even?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> auto_bootstrap is true by default. Ensure its set to true. On
>>>>>>> startup look at your logs for your auto_bootstrap value.  Look at the 
>>>>>>> node
>>>>>>> configuration line in your log file.
>>>>>>>
>>>>>>> Akhil
>>>>>>>
>>>>>>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> No, I didn't set it (left it at default value)
>>>>>>>>
>>>>>>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Did you make sure auto_bootstrap property is indeed set to [true]
>>>>>>>>> when you added the node?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>>>>>>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>>>>>>>> *To:* Akhil Mehra <akhilme...@gmail.com>
>>>>>>>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>;
>>>>>>>>> user@cassandra.apache.org
>>>>>>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing
>>>>>>>>> problem)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> not evenly, i have setup a new cluster with subset of data (around
>>>>>>>>> 5gb). using the configuration above I am getting these results
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Datacenter: datacenter1
>>>>>>>>>
>>>>>>>>> ===
>>>>>>>>>
>>>>>>>>> Status=Up/Down
>>>>>>>>>
>>>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>>>>
>>>>>>>>> --  Address  Load   Tokens   Owns (effective)  Host ID
>>>>>>>>>  Rack
>>>>>>>>>
>>>>>>>>> UN  10.128.2.1   4.86 GiB   256  44.9% 
>>>>>>>>> e4427611-c247-42ee-9404-371e177f5f17  rack1
>>>>>>>>>
>>>>>>>>> UN  10.128.2.10  725.03 MiB  256 55.1% 
>>>>>>>>> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>>>>>>>>
>>>>>>>>> is there anything else I can tweak/check to make the distribution
>>>>>>>>> even?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com>
&g

Re: Convert single node C* to cluster (rebalancing problem)

[Akhil Mehra]

.8%
>>>>> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
>>>>> UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
>>>>> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>>>>>
>>>>> Though maybe part of what you are experiencing can be cleared up by
>>>>> repair/compaction/cleanup. Also, what are your outputs when you call out
>>>>> specific keyspaces? Do the numbers get more even?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> auto_bootstrap is true by default. Ensure its set to true. On startup
>>>>>> look at your logs for your auto_bootstrap value.  Look at the node
>>>>>> configuration line in your log file.
>>>>>>
>>>>>> Akhil
>>>>>>
>>>>>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote:
>>>>>>
>>>>>>> No, I didn't set it (left it at default value)
>>>>>>>
>>>>>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Did you make sure auto_bootstrap property is indeed set to [true]
>>>>>>>> when you added the node?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>>>>>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>>>>>>> *To:* Akhil Mehra <akhilme...@gmail.com>
>>>>>>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>;
>>>>>>>> user@cassandra.apache.org
>>>>>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing
>>>>>>>> problem)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> not evenly, i have setup a new cluster with subset of data (around
>>>>>>>> 5gb). using the configuration above I am getting these results
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Datacenter: datacenter1
>>>>>>>>
>>>>>>>> ===
>>>>>>>>
>>>>>>>> Status=Up/Down
>>>>>>>>
>>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>>>
>>>>>>>> --  Address  Load   Tokens   Owns (effective)  Host ID 
>>>>>>>> Rack
>>>>>>>>
>>>>>>>> UN  10.128.2.1   4.86 GiB   256  44.9% 
>>>>>>>> e4427611-c247-42ee-9404-371e177f5f17  rack1
>>>>>>>>
>>>>>>>> UN  10.128.2.10  725.03 MiB  256 55.1% 
>>>>>>>> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>>>>>>>
>>>>>>>> is there anything else I can tweak/check to make the distribution
>>>>>>>> even?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> So now the data is evenly balanced in both nodes?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Refer to the following documentation to get a better understanding
>>>>>>>> of the roc_address and the broadcast_rpc_address https://
>>>>>>>> www.instaclustr.com/demystifying-cassandras-broadcast_address/
>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI=>.
>>>>>>>> I am surprised that your node started up with rpc_broadcast_address
>>>>>>>> set as this is an unsupported property. I am assuming you are
>>>>>>>> using Cassandra version 3.10.
>>>>>>>>

Re: Convert single node C* to cluster (rebalancing problem)

[Affan Syed]

John,

I am a co-worker with Junaid -- he is out sick, so just wanted to confirm
that one of your shots in the dark is correct. This is a RF of 1x

"CREATE KEYSPACE orion WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '1'}  AND durable_writes = true;"

However, how does the RF affect the redistribution of key/data?

Affan

- Affan

On Wed, Jun 14, 2017 at 1:16 AM, John Hughes <johnthug...@gmail.com> wrote:

> OP, I was just looking at your original numbers and I have some questions:
>
> 270GB on one node and 414KB on the other, but something close to 50/50 on
> "Owns(effective)".
> What replication factor are your keyspaces set up with? 1x or 2x or ??
>
> I would say you are seeing 50/50 because the tokens are allocated
> 50/50(others on the list please correct what are for me really just
> assumptions), but I would hazard a guess that your replication factor
> is still 1x, so it isn't moving anything around. Or your keyspace
> rplication is incorrect and isn't being distributed(I have had issues with
> the AWSMultiRegionSnitch and not getting the region correct[us-east vs
> us-east-1). It doesn't throw an error, but it doesn't work very well either
> =)
>
> Can you do a 'describe keyspace XXX' and show the first line(the CREATE
> KEYSPACE line).
>
> Mind you, these are all just shots in the dark from here.
>
> Cheers,
>
>
> On Tue, Jun 13, 2017 at 3:13 AM Junaid Nasir <jna...@an10.io> wrote:
>
>> Is the OP expecting a perfect 50%/50% split?
>>
>>
>> best result I got was 240gb/30gb split, which I think is not properly
>> balanced.
>>
>>
>>> Also, what are your outputs when you call out specific keyspaces? Do the
>>> numbers get more even?
>>
>>
>> i don't know what you mean by *call out specific key spaces?* can you
>> please explain that a bit.
>>
>>
>> If your schema is not modelled correctly you can easily end up unevenly
>>> distributed data.
>>
>>
>> I think that is the problem. initial 270gb data might not by modeled
>> correctly. I have run a lot of tests on 270gb data including downsizing it
>> to 5gb, they all resulted in same uneven distribution. I also tested a
>> dummy dataset of 2gb which was balanced evenly. coming from rdb, I didn't
>> give much thought to data modeling. can anyone please point me to some
>> resources regarding this problem.
>>
>> On Tue, Jun 13, 2017 at 3:24 AM, Akhil Mehra <akhilme...@gmail.com>
>> wrote:
>>
>>> Great point John.
>>>
>>> The OP should also note that data distribution also depends on your
>>> schema and incoming data profile.
>>>
>>> If your schema is not modelled correctly you can easily end up unevenly
>>> distributed data.
>>>
>>> Cheers,
>>> Akhil
>>>
>>> On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthug...@gmail.com>
>>> wrote:
>>>
>>>> Is the OP expecting a perfect 50%/50% split? That, to my experience, is
>>>> not going to happen, it is almost always shifted from a fraction of a
>>>> percent to a couple percent.
>>>>
>>>> Datacenter: eu-west
>>>> ===
>>>> Status=Up/Down
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>> --  AddressLoad   Tokens   Owns (effective)  Host ID
>>>> Rack
>>>> UN  XX.XX.XX.XX22.71 GiB  256  47.6%
>>>> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
>>>> UN  XX.XX.XX.XX  17.17 GiB  256  51.3%
>>>> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
>>>> UN  XX.XX.XX.XX  26.15 GiB  256  52.4%
>>>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
>>>> UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
>>>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
>>>> UN  XX.XX.XX.XX  24.39 GiB  256  49.8%
>>>> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
>>>> UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
>>>> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>>>>
>>>> Though maybe part of what you are experiencing can be cleared up by
>>>> repair/compaction/cleanup. Also, what are your outputs when you call out
>>>> specific keyspaces? Do the numbers get more even?
>>>>
>>>> Cheers,
>>>>
>>>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com>
>>>> wrote:
>>>>
>>>>> auto_bootstrap is true by default. Ensure its set to true

Re: Convert single node C* to cluster (rebalancing problem)

[John Hughes]

OP, I was just looking at your original numbers and I have some questions:

270GB on one node and 414KB on the other, but something close to 50/50 on
"Owns(effective)".
What replication factor are your keyspaces set up with? 1x or 2x or ??

I would say you are seeing 50/50 because the tokens are allocated
50/50(others on the list please correct what are for me really just
assumptions), but I would hazard a guess that your replication factor
is still 1x, so it isn't moving anything around. Or your keyspace
rplication is incorrect and isn't being distributed(I have had issues with
the AWSMultiRegionSnitch and not getting the region correct[us-east vs
us-east-1). It doesn't throw an error, but it doesn't work very well either
=)

Can you do a 'describe keyspace XXX' and show the first line(the CREATE
KEYSPACE line).

Mind you, these are all just shots in the dark from here.

Cheers,


On Tue, Jun 13, 2017 at 3:13 AM Junaid Nasir <jna...@an10.io> wrote:

> Is the OP expecting a perfect 50%/50% split?
>
>
> best result I got was 240gb/30gb split, which I think is not properly
> balanced.
>
>
>> Also, what are your outputs when you call out specific keyspaces? Do the
>> numbers get more even?
>
>
> i don't know what you mean by *call out specific key spaces?* can you
> please explain that a bit.
>
>
> If your schema is not modelled correctly you can easily end up unevenly
>> distributed data.
>
>
> I think that is the problem. initial 270gb data might not by modeled
> correctly. I have run a lot of tests on 270gb data including downsizing it
> to 5gb, they all resulted in same uneven distribution. I also tested a
> dummy dataset of 2gb which was balanced evenly. coming from rdb, I didn't
> give much thought to data modeling. can anyone please point me to some
> resources regarding this problem.
>
> On Tue, Jun 13, 2017 at 3:24 AM, Akhil Mehra <akhilme...@gmail.com> wrote:
>
>> Great point John.
>>
>> The OP should also note that data distribution also depends on your
>> schema and incoming data profile.
>>
>> If your schema is not modelled correctly you can easily end up unevenly
>> distributed data.
>>
>> Cheers,
>> Akhil
>>
>> On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthug...@gmail.com>
>> wrote:
>>
>>> Is the OP expecting a perfect 50%/50% split? That, to my experience, is
>>> not going to happen, it is almost always shifted from a fraction of a
>>> percent to a couple percent.
>>>
>>> Datacenter: eu-west
>>> ===
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  AddressLoad   Tokens   Owns (effective)  Host ID
>>>   Rack
>>> UN  XX.XX.XX.XX22.71 GiB  256  47.6%
>>> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
>>> UN  XX.XX.XX.XX  17.17 GiB  256  51.3%
>>> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
>>> UN  XX.XX.XX.XX  26.15 GiB  256  52.4%
>>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
>>> UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
>>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
>>> UN  XX.XX.XX.XX  24.39 GiB  256  49.8%
>>> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
>>> UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
>>> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>>>
>>> Though maybe part of what you are experiencing can be cleared up by
>>> repair/compaction/cleanup. Also, what are your outputs when you call out
>>> specific keyspaces? Do the numbers get more even?
>>>
>>> Cheers,
>>>
>>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com>
>>> wrote:
>>>
>>>> auto_bootstrap is true by default. Ensure its set to true. On startup
>>>> look at your logs for your auto_bootstrap value.  Look at the node
>>>> configuration line in your log file.
>>>>
>>>> Akhil
>>>>
>>>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote:
>>>>
>>>>> No, I didn't set it (left it at default value)
>>>>>
>>>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> wrote:
>>>>>
>>>>>> Did you make sure auto_bootstrap property is indeed set to [true]
>>>>>> when you added the node?
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>>>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>

Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

>
> Is the OP expecting a perfect 50%/50% split?


best result I got was 240gb/30gb split, which I think is not properly
balanced.


> Also, what are your outputs when you call out specific keyspaces? Do the
> numbers get more even?


i don't know what you mean by *call out specific key spaces?* can you
please explain that a bit.


If your schema is not modelled correctly you can easily end up unevenly
> distributed data.


I think that is the problem. initial 270gb data might not by modeled
correctly. I have run a lot of tests on 270gb data including downsizing it
to 5gb, they all resulted in same uneven distribution. I also tested a
dummy dataset of 2gb which was balanced evenly. coming from rdb, I didn't
give much thought to data modeling. can anyone please point me to some
resources regarding this problem.

On Tue, Jun 13, 2017 at 3:24 AM, Akhil Mehra <akhilme...@gmail.com> wrote:

> Great point John.
>
> The OP should also note that data distribution also depends on your schema
> and incoming data profile.
>
> If your schema is not modelled correctly you can easily end up unevenly
> distributed data.
>
> Cheers,
> Akhil
>
> On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthug...@gmail.com>
> wrote:
>
>> Is the OP expecting a perfect 50%/50% split? That, to my experience, is
>> not going to happen, it is almost always shifted from a fraction of a
>> percent to a couple percent.
>>
>> Datacenter: eu-west
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  AddressLoad   Tokens   Owns (effective)  Host ID
>>   Rack
>> UN  XX.XX.XX.XX22.71 GiB  256  47.6%
>> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
>> UN  XX.XX.XX.XX  17.17 GiB  256  51.3%
>> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
>> UN  XX.XX.XX.XX  26.15 GiB  256  52.4%
>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
>> UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
>> UN  XX.XX.XX.XX  24.39 GiB  256  49.8%
>> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
>> UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
>> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>>
>> Though maybe part of what you are experiencing can be cleared up by
>> repair/compaction/cleanup. Also, what are your outputs when you call out
>> specific keyspaces? Do the numbers get more even?
>>
>> Cheers,
>>
>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com> wrote:
>>
>>> auto_bootstrap is true by default. Ensure its set to true. On startup
>>> look at your logs for your auto_bootstrap value.  Look at the node
>>> configuration line in your log file.
>>>
>>> Akhil
>>>
>>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote:
>>>
>>>> No, I didn't set it (left it at default value)
>>>>
>>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> wrote:
>>>>
>>>>> Did you make sure auto_bootstrap property is indeed set to [true]
>>>>> when you added the node?
>>>>>
>>>>>
>>>>>
>>>>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>>>> *To:* Akhil Mehra <akhilme...@gmail.com>
>>>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>;
>>>>> user@cassandra.apache.org
>>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing problem)
>>>>>
>>>>>
>>>>>
>>>>> not evenly, i have setup a new cluster with subset of data (around
>>>>> 5gb). using the configuration above I am getting these results
>>>>>
>>>>>
>>>>>
>>>>> Datacenter: datacenter1
>>>>>
>>>>> ===
>>>>>
>>>>> Status=Up/Down
>>>>>
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>
>>>>> --  Address  Load   Tokens   Owns (effective)  Host ID 
>>>>> Rack
>>>>>
>>>>> UN  10.128.2.1   4.86 GiB   256  44.9% 
>>>>> e4427611-c247-42ee-9404-371e177f5f17  rack1
>>>>>
>>>>> UN  10.128.2.10  725.03 MiB  256 55.1% 
>>>>> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>>>>
>>>>> is there anythi

Re: Convert single node C* to cluster (rebalancing problem)

[Akhil Mehra]

Great point John.

The OP should also note that data distribution also depends on your schema
and incoming data profile.

If your schema is not modelled correctly you can easily end up unevenly
distributed data.

Cheers,
Akhil

On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthug...@gmail.com> wrote:

> Is the OP expecting a perfect 50%/50% split? That, to my experience, is
> not going to happen, it is almost always shifted from a fraction of a
> percent to a couple percent.
>
> Datacenter: eu-west
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID
> Rack
> UN  XX.XX.XX.XX22.71 GiB  256  47.6%
> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
> UN  XX.XX.XX.XX  17.17 GiB  256  51.3%
> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
> UN  XX.XX.XX.XX  26.15 GiB  256  52.4%
> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
> UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
> UN  XX.XX.XX.XX  24.39 GiB  256  49.8%
> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
> UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>
> Though maybe part of what you are experiencing can be cleared up by
> repair/compaction/cleanup. Also, what are your outputs when you call out
> specific keyspaces? Do the numbers get more even?
>
> Cheers,
>
> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com> wrote:
>
>> auto_bootstrap is true by default. Ensure its set to true. On startup
>> look at your logs for your auto_bootstrap value.  Look at the node
>> configuration line in your log file.
>>
>> Akhil
>>
>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote:
>>
>>> No, I didn't set it (left it at default value)
>>>
>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> wrote:
>>>
>>>> Did you make sure auto_bootstrap property is indeed set to [true] when
>>>> you added the node?
>>>>
>>>>
>>>>
>>>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>>> *To:* Akhil Mehra <akhilme...@gmail.com>
>>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>;
>>>> user@cassandra.apache.org
>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing problem)
>>>>
>>>>
>>>>
>>>> not evenly, i have setup a new cluster with subset of data (around
>>>> 5gb). using the configuration above I am getting these results
>>>>
>>>>
>>>>
>>>> Datacenter: datacenter1
>>>>
>>>> ===
>>>>
>>>> Status=Up/Down
>>>>
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>
>>>> --  Address  Load   Tokens   Owns (effective)  Host ID Rack
>>>>
>>>> UN  10.128.2.1   4.86 GiB   256  44.9% 
>>>> e4427611-c247-42ee-9404-371e177f5f17  rack1
>>>>
>>>> UN  10.128.2.10  725.03 MiB  256 55.1% 
>>>> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>>>
>>>> is there anything else I can tweak/check to make the distribution even?
>>>>
>>>>
>>>>
>>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com>
>>>> wrote:
>>>>
>>>> So now the data is evenly balanced in both nodes?
>>>>
>>>>
>>>>
>>>> Refer to the following documentation to get a better understanding of
>>>> the roc_address and the broadcast_rpc_address https://
>>>> www.instaclustr.com/demystifying-cassandras-broadcast_address/
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI=>.
>>>> I am surprised that your node started up with rpc_broadcast_address
>>>> set as this is an unsupported property. I am assuming you are using
>>>> Cassandra version 3.10.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Akhil
>>>>
>>>>
>>>>
>>>> On 2/06/2017, at 11:0

Re: Convert single node C* to cluster (rebalancing problem)

[John Hughes]

Is the OP expecting a perfect 50%/50% split? That, to my experience, is not
going to happen, it is almost always shifted from a fraction of a percent
to a couple percent.

Datacenter: eu-west
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens   Owns (effective)  Host ID
Rack
UN  XX.XX.XX.XX22.71 GiB  256  47.6%
57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
UN  XX.XX.XX.XX  17.17 GiB  256  51.3%
d2a65c51-087d-48de-ae1f-a41142eb148d  1b
UN  XX.XX.XX.XX  26.15 GiB  256  52.4%
acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
UN  XX.XX.XX.XX  24.39 GiB  256  49.8%
fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
bdc597c0-718c-4ef6-b3ef-7785110a9923  1b

Though maybe part of what you are experiencing can be cleared up by
repair/compaction/cleanup. Also, what are your outputs when you call out
specific keyspaces? Do the numbers get more even?

Cheers,

On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com> wrote:

> auto_bootstrap is true by default. Ensure its set to true. On startup look
> at your logs for your auto_bootstrap value.  Look at the node configuration
> line in your log file.
>
> Akhil
>
> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote:
>
>> No, I didn't set it (left it at default value)
>>
>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> wrote:
>>
>>> Did you make sure auto_bootstrap property is indeed set to [true] when
>>> you added the node?
>>>
>>>
>>>
>>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>> *To:* Akhil Mehra <akhilme...@gmail.com>
>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>; user@cassandra.apache.org
>>> *Subject:* Re: Convert single node C* to cluster (rebalancing problem)
>>>
>>>
>>>
>>> not evenly, i have setup a new cluster with subset of data (around 5gb).
>>> using the configuration above I am getting these results
>>>
>>>
>>>
>>> Datacenter: datacenter1
>>>
>>> ===
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> --  Address  Load   Tokens   Owns (effective)  Host ID Rack
>>>
>>> UN  10.128.2.1   4.86 GiB   256  44.9% 
>>> e4427611-c247-42ee-9404-371e177f5f17  rack1
>>>
>>> UN  10.128.2.10  725.03 MiB  256 55.1% 
>>> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>>
>>> is there anything else I can tweak/check to make the distribution even?
>>>
>>>
>>>
>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com>
>>> wrote:
>>>
>>> So now the data is evenly balanced in both nodes?
>>>
>>>
>>>
>>> Refer to the following documentation to get a better understanding of
>>> the roc_address and the broadcast_rpc_address
>>> https://www.instaclustr.com/demystifying-cassandras-broadcast_address/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI=>.
>>> I am surprised that your node started up with rpc_broadcast_address set as
>>> this is an unsupported property. I am assuming you are using Cassandra
>>> version 3.10.
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Akhil
>>>
>>>
>>>
>>> On 2/06/2017, at 11:06 PM, Junaid Nasir <jna...@an10.io> wrote:
>>>
>>>
>>>
>>> I am able to get it working. I added a new node with following changes
>>>
>>> #rpc_address:0.0.0.0
>>>
>>> rpc_address: 10.128.1.11
>>>
>>> #rpc_broadcast_address:10.128.1.11
>>>
>>> rpc_address was set to 0.0.0.0, (I ran into a problem previously
>>> regarding remote connection and made these changes
>>> https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_12236898_apache-2Dcassandra-2Dremote-2Daccess=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1G

Re: Convert single node C* to cluster (rebalancing problem)

[Akhil Mehra]

auto_bootstrap is true by default. Ensure its set to true. On startup look
at your logs for your auto_bootstrap value.  Look at the node configuration
line in your log file.

Akhil

On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote:

> No, I didn't set it (left it at default value)
>
> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> wrote:
>
>> Did you make sure auto_bootstrap property is indeed set to [true] when
>> you added the node?
>>
>>
>>
>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>> *Sent:* Monday, June 05, 2017 6:29 AM
>> *To:* Akhil Mehra <akhilme...@gmail.com>
>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>; user@cassandra.apache.org
>> *Subject:* Re: Convert single node C* to cluster (rebalancing problem)
>>
>>
>>
>> not evenly, i have setup a new cluster with subset of data (around 5gb).
>> using the configuration above I am getting these results
>>
>>
>>
>> Datacenter: datacenter1
>>
>> ===
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  Address  Load   Tokens   Owns (effective)  Host ID Rack
>>
>> UN  10.128.2.1   4.86 GiB   256  44.9% 
>> e4427611-c247-42ee-9404-371e177f5f17  rack1
>>
>> UN  10.128.2.10  725.03 MiB  256 55.1% 
>> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>
>> is there anything else I can tweak/check to make the distribution even?
>>
>>
>>
>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com> wrote:
>>
>> So now the data is evenly balanced in both nodes?
>>
>>
>>
>> Refer to the following documentation to get a better understanding of the
>> roc_address and the broadcast_rpc_address https://
>> www.instaclustr.com/demystifying-cassandras-broadcast_address/
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI=>.
>> I am surprised that your node started up with rpc_broadcast_address set as
>> this is an unsupported property. I am assuming you are using Cassandra
>> version 3.10.
>>
>>
>>
>>
>>
>> Regards,
>>
>> Akhil
>>
>>
>>
>> On 2/06/2017, at 11:06 PM, Junaid Nasir <jna...@an10.io> wrote:
>>
>>
>>
>> I am able to get it working. I added a new node with following changes
>>
>> #rpc_address:0.0.0.0
>>
>> rpc_address: 10.128.1.11
>>
>> #rpc_broadcast_address:10.128.1.11
>>
>> rpc_address was set to 0.0.0.0, (I ran into a problem previously
>> regarding remote connection and made these changes
>> https://stackoverflow.com/questions/12236898/apache-cassandr
>> a-remote-access
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_12236898_apache-2Dcassandra-2Dremote-2Daccess=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=oj8BCLiyBDqqVQNqfGohFeujtqlzAkd-uwS878d4mg4=>
>> )
>>
>>
>>
>> should it be happening?
>>
>>
>>
>> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin <vla...@winguzone.com>
>> wrote:
>>
>> Did you run "nodetool cleanup" on first node after second was
>> bootstrapped? It should clean rows not belonging to node after tokens
>> changed.
>>
>>
>>
>> Best regards, Vladimir Yudovin,
>>
>> *Winguzone
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com_-3Ffrom-3Dlist=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=Q1M5YRAsw0iUQKOIulEmO72RhdENQCRhpqZSjgxxHos=>
>> - Cloud Cassandra Hosting*
>>
>>
>>
>>
>>
>>  On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir <jna...@an10.io
>> <jna...@an10.io>>* wrote 
>>
>>
>>
>> Cassandra ensure that adding or removing nodes are very easy and that
>> load is balanced between nodes when a change is made. but it's not working
>> in my case.
>>
>> I have a single node C* deployment (with 270 GB of data) and want to load
>> balance the data on multiple nodes, I followed this guide
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_cassandra_2.1_cassandra_operations_ops-5Fadd-5Fnode-5Fto-5Fcluster-5Ft.h

Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

No, I didn't set it (left it at default value)

On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> wrote:

> Did you make sure auto_bootstrap property is indeed set to [true] when
> you added the node?
>
>
>
> *From:* Junaid Nasir [mailto:jna...@an10.io]
> *Sent:* Monday, June 05, 2017 6:29 AM
> *To:* Akhil Mehra <akhilme...@gmail.com>
> *Cc:* Vladimir Yudovin <vla...@winguzone.com>; user@cassandra.apache.org
> *Subject:* Re: Convert single node C* to cluster (rebalancing problem)
>
>
>
> not evenly, i have setup a new cluster with subset of data (around 5gb).
> using the configuration above I am getting these results
>
>
>
> Datacenter: datacenter1
>
> ===
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> --  Address  Load   Tokens   Owns (effective)  Host ID Rack
>
> UN  10.128.2.1   4.86 GiB   256  44.9% 
> e4427611-c247-42ee-9404-371e177f5f17  rack1
>
> UN  10.128.2.10  725.03 MiB  256 55.1% 
> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>
> is there anything else I can tweak/check to make the distribution even?
>
>
>
> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com> wrote:
>
> So now the data is evenly balanced in both nodes?
>
>
>
> Refer to the following documentation to get a better understanding of the
> roc_address and the broadcast_rpc_address https://www.instaclustr.com/
> demystifying-cassandras-broadcast_address/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI=>.
> I am surprised that your node started up with rpc_broadcast_address set as
> this is an unsupported property. I am assuming you are using Cassandra
> version 3.10.
>
>
>
>
>
> Regards,
>
> Akhil
>
>
>
> On 2/06/2017, at 11:06 PM, Junaid Nasir <jna...@an10.io> wrote:
>
>
>
> I am able to get it working. I added a new node with following changes
>
> #rpc_address:0.0.0.0
>
> rpc_address: 10.128.1.11
>
> #rpc_broadcast_address:10.128.1.11
>
> rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding
> remote connection and made these changes https://stackoverflow.com/
> questions/12236898/apache-cassandra-remote-access
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_12236898_apache-2Dcassandra-2Dremote-2Daccess=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=oj8BCLiyBDqqVQNqfGohFeujtqlzAkd-uwS878d4mg4=>
> )
>
>
>
> should it be happening?
>
>
>
> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin <vla...@winguzone.com>
> wrote:
>
> Did you run "nodetool cleanup" on first node after second was
> bootstrapped? It should clean rows not belonging to node after tokens
> changed.
>
>
>
> Best regards, Vladimir Yudovin,
>
> *Winguzone
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com_-3Ffrom-3Dlist=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=Q1M5YRAsw0iUQKOIulEmO72RhdENQCRhpqZSjgxxHos=>
> - Cloud Cassandra Hosting*
>
>
>
>
>
>  On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir <jna...@an10.io
> <jna...@an10.io>>* wrote 
>
>
>
> Cassandra ensure that adding or removing nodes are very easy and that load
> is balanced between nodes when a change is made. but it's not working in my
> case.
>
> I have a single node C* deployment (with 270 GB of data) and want to load
> balance the data on multiple nodes, I followed this guide
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_cassandra_2.1_cassandra_operations_ops-5Fadd-5Fnode-5Fto-5Fcluster-5Ft.html=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=xnLuX4qqSZxVuY6Gz2NXRdc6TG8J7at9kDhkhBnWpnM=>
>
>
> `nodetool status` shows 2 nodes but load is not balanced between them
>
> Datacenter: dc1
>
> ===
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> --  Address  Load   Tokens   Owns (effective)  Host IDRack
>
> UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
>
> UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>
> I also ran 'nodetool repair' on new node but result is same. any pointers
> would be appreciated :)
>
>
>
> conf file of new node
>
> cluster_name: 'cluster1'
>
>  - seeds: "10.128.0.7"
> num_tokens: 256
>
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Thanks,
>
> Junaid
>
>
>
>
>
>
>
>
>

RE: Convert single node C* to cluster (rebalancing problem)

[ZAIDI, ASAD A]

Did you make sure auto_bootstrap property is indeed set to [true] when you 
added the node?

From: Junaid Nasir [mailto:jna...@an10.io]
Sent: Monday, June 05, 2017 6:29 AM
To: Akhil Mehra <akhilme...@gmail.com>
Cc: Vladimir Yudovin <vla...@winguzone.com>; user@cassandra.apache.org
Subject: Re: Convert single node C* to cluster (rebalancing problem)

not evenly, i have setup a new cluster with subset of data (around 5gb). using 
the configuration above I am getting these results


Datacenter: datacenter1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address  Load   Tokens   Owns (effective)  Host ID Rack

UN  10.128.2.1   4.86 GiB   256  44.9% 
e4427611-c247-42ee-9404-371e177f5f17  rack1

UN  10.128.2.10  725.03 MiB  256 55.1% 
690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
is there anything else I can tweak/check to make the distribution even?

On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra 
<akhilme...@gmail.com<mailto:akhilme...@gmail.com>> wrote:
So now the data is evenly balanced in both nodes?

Refer to the following documentation to get a better understanding of the 
roc_address and the broadcast_rpc_address 
https://www.instaclustr.com/demystifying-cassandras-broadcast_address/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI=>.
 I am surprised that your node started up with rpc_broadcast_address set as 
this is an unsupported property. I am assuming you are using Cassandra version 
3.10.


Regards,
Akhil

On 2/06/2017, at 11:06 PM, Junaid Nasir <jna...@an10.io<mailto:jna...@an10.io>> 
wrote:

I am able to get it working. I added a new node with following changes

#rpc_address:0.0.0.0

rpc_address: 10.128.1.11

#rpc_broadcast_address:10.128.1.11
rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding 
remote connection and made these changes 
https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access<https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_12236898_apache-2Dcassandra-2Dremote-2Daccess=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=oj8BCLiyBDqqVQNqfGohFeujtqlzAkd-uwS878d4mg4=>)

should it be happening?

On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin 
<vla...@winguzone.com<mailto:vla...@winguzone.com>> wrote:
Did you run "nodetool cleanup" on first node after second was bootstrapped? It 
should clean rows not belonging to node after tokens changed.

Best regards, Vladimir Yudovin,
Winguzone<https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com_-3Ffrom-3Dlist=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=Q1M5YRAsw0iUQKOIulEmO72RhdENQCRhpqZSjgxxHos=>
 - Cloud Cassandra Hosting


 On Wed, 31 May 2017 03:55:54 -0400 Junaid Nasir 
<jna...@an10.io<mailto:jna...@an10.io>> wrote 

Cassandra ensure that adding or removing nodes are very easy and that load is 
balanced between nodes when a change is made. but it's not working in my case.
I have a single node C* deployment (with 270 GB of data) and want to load 
balance the data on multiple nodes, I followed this 
guide<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_cassandra_2.1_cassandra_operations_ops-5Fadd-5Fnode-5Fto-5Fcluster-5Ft.html=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=xnLuX4qqSZxVuY6Gz2NXRdc6TG8J7at9kDhkhBnWpnM=>
`nodetool status` shows 2 nodes but load is not balanced between them

Datacenter: dc1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address  Load   Tokens   Owns (effective)  Host IDRack

UN  10.128.0.7   270.75 GiB  256  48.6%
1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1

UN  10.128.0.14  414.36 KiB  256  51.4%
66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
I also ran 'nodetool repair' on new node but result is same. any pointers would 
be appreciated :)

conf file of new node

cluster_name: 'cluster1'

 - seeds: "10.128.0.7"
num_tokens: 256

endpoint_snitch: GossipingPropertyFileSnitch
Thanks,
Junaid

Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

not evenly, i have setup a new cluster with subset of data (around 5gb).
using the configuration above I am getting these results

Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address  Load   Tokens   Owns (effective)  Host ID Rack
UN  10.128.2.1   4.86 GiB   256  44.9%
e4427611-c247-42ee-9404-371e177f5f17  rack1
UN  10.128.2.10  725.03 MiB  256 55.1%
690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1

is there anything else I can tweak/check to make the distribution even?

On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra  wrote:

> So now the data is evenly balanced in both nodes?
>
> Refer to the following documentation to get a better understanding of the
> roc_address and the broadcast_rpc_address https://www.instaclustr.com/
> demystifying-cassandras-broadcast_address/. I am surprised that your node
> started up with rpc_broadcast_address set as this is an unsupported
> property. I am assuming you are using Cassandra version 3.10.
>
>
> Regards,
> Akhil
>
> On 2/06/2017, at 11:06 PM, Junaid Nasir  wrote:
>
> I am able to get it working. I added a new node with following changes
>
> #rpc_address:0.0.0.0
> rpc_address: 10.128.1.11
> #rpc_broadcast_address:10.128.1.11
>
> rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding
> remote connection and made these changes https://stackoverflow.com/
> questions/12236898/apache-cassandra-remote-access)
>
> should it be happening?
>
> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin 
> wrote:
>
>> Did you run "nodetool cleanup" on first node after second was
>> bootstrapped? It should clean rows not belonging to node after tokens
>> changed.
>>
>> Best regards, Vladimir Yudovin,
>> *Winguzone  - Cloud Cassandra Hosting*
>>
>>
>>  On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir > >* wrote 
>>
>> Cassandra ensure that adding or removing nodes are very easy and that
>> load is balanced between nodes when a change is made. but it's not working
>> in my case.
>> I have a single node C* deployment (with 270 GB of data) and want to load
>> balance the data on multiple nodes, I followed this guide
>> 
>>
>> `nodetool status` shows 2 nodes but load is not balanced between them
>>
>> Datacenter: dc1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens   Owns (effective)  Host IDRack
>> UN  10.128.0.7   270.75 GiB  256  48.6%
>> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
>> UN  10.128.0.14  414.36 KiB  256  51.4%
>> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>>
>> I also ran 'nodetool repair' on new node but result is same. any pointers
>> would be appreciated :)
>>
>> conf file of new node
>>
>> cluster_name: 'cluster1'
>>  - seeds: "10.128.0.7"
>> num_tokens: 256
>> endpoint_snitch: GossipingPropertyFileSnitch
>>
>> Thanks,
>> Junaid
>>
>>
>>
>
>

Re: Convert single node C* to cluster (rebalancing problem)

[Akhil Mehra]

So now the data is evenly balanced in both nodes? 

Refer to the following documentation to get a better understanding of the 
roc_address and the broadcast_rpc_address 
https://www.instaclustr.com/demystifying-cassandras-broadcast_address/ 
. I am 
surprised that your node started up with rpc_broadcast_address set as this is 
an unsupported property. I am assuming you are using Cassandra version 3.10.


Regards,
Akhil

> On 2/06/2017, at 11:06 PM, Junaid Nasir  wrote:
> 
> I am able to get it working. I added a new node with following changes
> #rpc_address:0.0.0.0
> rpc_address: 10.128.1.11
> #rpc_broadcast_address:10.128.1.11
> rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding 
> remote connection and made these changes 
> https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access 
> )
>  
> 
> should it be happening?
> 
> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin  > wrote:
> Did you run "nodetool cleanup" on first node after second was bootstrapped? 
> It should clean rows not belonging to node after tokens changed.
> 
> Best regards, Vladimir Yudovin, 
> Winguzone  - Cloud Cassandra Hosting
> 
> 
>  On Wed, 31 May 2017 03:55:54 -0400 Junaid Nasir  > wrote 
> 
> Cassandra ensure that adding or removing nodes are very easy and that load is 
> balanced between nodes when a change is made. but it's not working in my case.
> I have a single node C* deployment (with 270 GB of data) and want to load 
> balance the data on multiple nodes, I followed this guide 
> 
>  
> `nodetool status` shows 2 nodes but load is not balanced between them
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens   Owns (effective)  Host IDRack
> UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
> UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
> I also ran 'nodetool repair' on new node but result is same. any pointers 
> would be appreciated :)
> 
> conf file of new node
> cluster_name: 'cluster1'
>  - seeds: "10.128.0.7"
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
> Thanks,
> Junaid
> 
> 

Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

I am able to get it working. I added a new node with following changes

#rpc_address:0.0.0.0
rpc_address: 10.128.1.11
#rpc_broadcast_address:10.128.1.11

rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding
remote connection and made these changes
https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access
)

should it be happening?

On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin 
wrote:

> Did you run "nodetool cleanup" on first node after second was
> bootstrapped? It should clean rows not belonging to node after tokens
> changed.
>
> Best regards, Vladimir Yudovin,
> *Winguzone  - Cloud Cassandra Hosting*
>
>
>  On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir  >* wrote 
>
> Cassandra ensure that adding or removing nodes are very easy and that load
> is balanced between nodes when a change is made. but it's not working in my
> case.
> I have a single node C* deployment (with 270 GB of data) and want to load
> balance the data on multiple nodes, I followed this guide
> 
>
> `nodetool status` shows 2 nodes but load is not balanced between them
>
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens   Owns (effective)  Host IDRack
> UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
> UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>
> I also ran 'nodetool repair' on new node but result is same. any pointers
> would be appreciated :)
>
> conf file of new node
>
> cluster_name: 'cluster1'
>  - seeds: "10.128.0.7"
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Thanks,
> Junaid
>
>
>

Re: Convert single node C* to cluster (rebalancing problem)

[Vladimir Yudovin]

Did you run "nodetool cleanup" on first node after second was bootstrapped? It 
should clean rows not belonging to node after tokens changed.



Best regards, Vladimir Yudovin, 

Winguzone - Cloud Cassandra Hosting






 On Wed, 31 May 2017 03:55:54 -0400 Junaid Nasir jna...@an10.io 
wrote 




Cassandra ensure that adding or removing nodes are very easy and that load is 
balanced between nodes when a change is made. but it's not working in my case.

I have a single node C* deployment (with 270 GB of data) and want to load 
balance the data on multiple nodes, I followed this guide 

`nodetool status` shows 2 nodes but load is not balanced between them

Datacenter: dc1 === Status=Up/Down |/ 
State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) 
Host ID Rack UN 10.128.0.7 270.75 GiB 256 48.6% 
1a3f6faa-4376-45a8-9c20-11480ae5664c rack1 UN 10.128.0.14 414.36 KiB 256 51.4% 
66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1

I also ran 'nodetool repair' on new node but result is same. any pointers would 
be appreciated :)



conf file of new node

cluster_name: 'cluster1' - seeds: "10.128.0.7"
num_tokens: 256 endpoint_snitch: GossipingPropertyFileSnitch
Thanks,

Junaid

Re: Convert single node C* to cluster (rebalancing problem)

[Akhil Mehra]

When you bootstrapped the node for the first time did you see log similar to 
the following:
INFO  [main] 2017-06-01 07:19:45,199 StorageService.java:1435 - JOINING: 
waiting for schema information to complete
INFO  [main] 2017-06-01 07:19:45,250 StorageService.java:1435 - JOINING: schema 
complete, ready to bootstrap
INFO  [main] 2017-06-01 07:19:45,251 StorageService.java:1435 - JOINING: 
waiting for pending range calculation
INFO  [main] 2017-06-01 07:19:45,251 StorageService.java:1435 - JOINING: 
calculation complete, ready to bootstrap
INFO  [main] 2017-06-01 07:19:45,251 StorageService.java:1435 - JOINING: 
getting bootstrap token
INFO  [main] 2017-06-01 07:19:45,341 StorageService.java:1435 - JOINING: 
sleeping 3 ms for pending range setup
INFO  [main] 2017-06-01 07:20:15,342 StorageService.java:1435 - JOINING: 
Starting to bootstrap...
INFO  [main] 2017-06-01 07:20:15,562 StreamResultFuture.java:90 - [Stream 
#c219d430-469a-11e7-8af3-81773e2a69ae] Executing streaming plan for Bootstrap
INFO  [StreamConnectionEstablisher:1] 2017-06-01 07:20:15,568 
StreamSession.java:266 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] 
Starting streaming to /172.20.0.3
INFO  [StreamConnectionEstablisher:1] 2017-06-01 07:20:15,591 
StreamCoordinator.java:264 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae, 
ID#0] Beginning stream session with /172.20.0.3
INFO  [STREAM-IN-/172.20.0.3:7000] 2017-06-01 07:20:16,369 
StreamResultFuture.java:173 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae 
ID#0] Prepare completed. Receiving 4 files(4.046MiB), sending 0 files(0.000KiB)
INFO  [StreamReceiveTask:1] 2017-06-01 07:20:32,489 StreamResultFuture.java:187 
- [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] Session with /172.20.0.3 is 
complete
INFO  [StreamReceiveTask:1] 2017-06-01 07:20:32,529 StreamResultFuture.java:219 
- [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] All sessions completed
INFO  [StreamReceiveTask:1] 2017-06-01 07:20:32,535 StorageService.java:1491 - 
Bootstrap completed! for the tokens [2170857698622202367, 8517072504425343717, 
-3254771037524187900, -1597835042001935502, 6878847904605480741, 
6816916215341820068, 6291189887494617640, -6855333019196580358, 
-6353317035112065873, 8838974905234547016, 853981057438397, 
-2357950949959511387, 1242077960532340887, 2914039668080386735, 
3548015300105653368, 8973388453035242795, -2325235809399362967, 
-7078812010537656277, 768585495224455336, 7153512700965912517, 
8625819392009074153, -6138302849441936958, -2594051958993427953, 
-735827743339795655, -8202727571538912843, 2180751358288507888, 
-7872842094207074012, -2926504780761300623, -3197260822146229664, 
3411052656191450941, -9049284186987733291, -157882351668930258, 
454637839762305232, -2305675997627138050, 5785282040753174988, 
8604531769609599767, 4363117061247143957, -7255854383313210529, 
-3497611663121502480, -6788457421774336480, -7809767930173770420, 
6591540654522244365, 1773283733607350132, 134776973669066, 
-7242556233623424655, -1552552727731631642, -1226243976028310059, 
-8221762326275074149, -7963893314043006091, -850542197910474448, 
4219437099703910566, -8039365343972054221, 7756456412568178996, 
4057327843751741693, 7155628666873897485, -483058846775660782, 
6968839681845709305, 6396337738827005745, -5285173481531605912, 
7254663657455123842, 871654822989271789, -604574593420741277, 
-244646170484127, -3707613591745746278, -26727542030118959, 
-7190990795521107837, 5388348291571480415, 4249499356533972018, 
82469082189512791, -6389351372873749061, 5138413916027470955, 
2542233707258091740, -4057927973990056143, 552933169018893618, 
-8237860380097407047, 6917383508758068288, 543382311932406672, 
-5671560690999322491, -1240369858424929757, 7394536427227616773, 
4716882285905136652, 8260705434779371419, 3259812719139852593, 
-73864539388331289, -3573980475038135246, -1047139059901238511, 
-1734886021153324482, 8674873751672827600, 3564384074427511950, 
2754071903665103098, -1230493021099846761, -2731315467436512731, 
-7845984767828231726, -8082165594257396645, -2298177264815779081, 
-364542048544165, 9142633389925493379, 7206663288804675578, 
2305939212045070856, -5101738026249032246, 6268847697773786891, 
5903922100677671597, -2001787466557152206, 1318502870562311928, 
5784020265166141829, 5385229217299505171, 6010414616247875068, 
-8080602674779008196, -9189764569651551963, -8969124116887255329, 
-9040482343274988119, -8575947267671214955, -1786409930636352174, 
-757203989676123224, -6640569567328853730, 8431839804447545665, 
6781635966829972979, -8328382509754233304, -3181089993114819214, 
3243262023331941781, 4213737472390389773, -4046361821170607634, 
8877904009116429296, -6931048276693039052, 4838006612846181604, 
-5561480934050473057, -470112649587309682, 3175935810873308999, 
-1693695808908080717, -3753035103371291265, -260741269584337, 
-8454963020263227780, 2037428931895594762, 1158209127301347406, 
-8092787384269386871, -7741092217712244823, 

Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

sorry that was an old notetool status output (old cluster with a few
configuration changes). new configuration is

--  Address  Load   Tokens   Owns (effective)  Host ID
  Rack
UN  10.128.1.2   270.75 GiB  256  52.6%
e4427611-c247-42ee-9404-371e177f5f17  rack1
UN  10.128.1.10  388.52 KiB  256  47.4%
7059e622-7861-495b-a69f-5a68876364dd  rack1

10.128.1.2 is seed node, and no other C* node is running on the network.
data_file_directories on seed node is not in default location, it's on
another disk. if that helps

On Thu, Jun 1, 2017 at 4:49 AM, Akhil Mehra  wrote:

> Hi Junaid,
>
> I noticed in the log files that data is being streamed from/to 10.128.1.2
> but the address of your original node is 10.128.0.7.
>
> Are there any other Cassandra nodes on your local network the same cluster
> name.
>
> Regards,
> Akhil
>
> > 10.128.1.2
> > On 31/05/2017, at 10:15 PM, Junaid Nasir  wrote:
> >
> > after nodetool repair on new node following log appears in cassandra log
> >
> > INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496
> StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> Starting streaming to /10.128.1.2
> > INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501
> StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac,
> ID#0] Beginning stream session with /10.128.1.2
> > INFO  [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545
> StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac
> ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0
> files(0.000KiB)
> > INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,705
> StreamResultFuture.java:187 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> Session with /10.128.1.2 is complete
> > INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,707
> StreamResultFuture.java:219 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> All sessions completed
> >
> >
> >
> > On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir  wrote:
> > Cassandra version is 3.10, and yes its not a production server. i have
> seen some warnings in logs saying token exist on both servers. other than
> that nothing. if you need any more settings/details please ask. thank you
> for your time
> >
> >
> > On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
> > On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:
> > Cassandra ensure that adding or removing nodes are very easy and that
> load is balanced between nodes when a change is made. but it's not working
> in my case.
> > I have a single node C* deployment (with 270 GB of data) and want to
> load balance the data on multiple nodes,
> >
> > I guess it's fair to assume this is not a production "cluster"?
> >
> > I followed this guide
> > `nodetool status` shows 2 nodes but load is not balanced between them
> > Datacenter: dc1
> > ===
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/M
> > oving
> > --  Address  Load   Tokens   Owns (effective)  Host ID
> Rack
> > UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480a
> > e5664c  rack1
> > UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a
> > 199b41  rack1
> > I can imagine you are referring to Load column here, which shows only
> 400 KB for the new node.  Did the newly added node actually bootstrap?  Are
> there any error/warning messages in the Cassandra log?
> >
> > I also ran 'nodetool repair' on new node but result is same. any
> pointers would be appreciated :)
> >
> > Hm, this is not expected.  Even if the node didn't bootstrap, repair
> should have streamed the data it is responsible for.
> >
> > conf file of new node
> > cluster_name: 'cluster1'
> >  - seeds: "10.128.0.7"
> >
> > num_tokens: 256
> > endpoint_snitch: GossipingPropertyFileSnitch
> >
> > I see that you've tried to add the new node as a normal (non-seed) node,
> which supposed to work and it should have bootstrapped itself.  Which
> version of Cassandra is it exactly?
> >
> > --
> > Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707
> >
> >
> >
>
>

Re: Convert single node C* to cluster (rebalancing problem)

[Akhil Mehra]

Hi Junaid,

I noticed in the log files that data is being streamed from/to 10.128.1.2 but 
the address of your original node is 10.128.0.7.

Are there any other Cassandra nodes on your local network the same cluster 
name. 

Regards,
Akhil

> 10.128.1.2
> On 31/05/2017, at 10:15 PM, Junaid Nasir  wrote:
> 
> after nodetool repair on new node following log appears in cassandra log
> 
> INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496 
> StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> Starting streaming to /10.128.1.2
> INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501 
> StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac, 
> ID#0] Beginning stream session with /10.128.1.2
> INFO  [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545 
> StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac 
> ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0 
> files(0.000KiB)
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,705 
> StreamResultFuture.java:187 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> Session with /10.128.1.2 is complete
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,707 
> StreamResultFuture.java:219 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> All sessions completed
> 
> 
> 
> On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir  wrote:
> Cassandra version is 3.10, and yes its not a production server. i have seen 
> some warnings in logs saying token exist on both servers. other than that 
> nothing. if you need any more settings/details please ask. thank you for your 
> time 
> 
> 
> On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin 
>  wrote:
> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:
> Cassandra ensure that adding or removing nodes are very easy and that load is 
> balanced between nodes when a change is made. but it's not working in my case.
> I have a single node C* deployment (with 270 GB of data) and want to load 
> balance the data on multiple nodes,
> 
> I guess it's fair to assume this is not a production "cluster"?
>  
> I followed this guide 
> `nodetool status` shows 2 nodes but load is not balanced between them
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/M
> oving
> --  Address  Load   Tokens   Owns (effective)  Host IDRack
> UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480a
> e5664c  rack1
> UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a
> 199b41  rack1
> I can imagine you are referring to Load column here, which shows only 400 KB 
> for the new node.  Did the newly added node actually bootstrap?  Are there 
> any error/warning messages in the Cassandra log?
>  
> I also ran 'nodetool repair' on new node but result is same. any pointers 
> would be appreciated :)
> 
> Hm, this is not expected.  Even if the node didn't bootstrap, repair should 
> have streamed the data it is responsible for.
> 
> conf file of new node
> cluster_name: 'cluster1'
>  - seeds: "10.128.0.7"
> 
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
> 
> I see that you've tried to add the new node as a normal (non-seed) node, 
> which supposed to work and it should have bootstrapped itself.  Which version 
> of Cassandra is it exactly?
> 
> -- 
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 
> 127-59-707
> 
> 
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

after nodetool repair on new node following log appears in cassandra log

INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496
> StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> Starting streaming to /10.128.1.2
> INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501
> StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac,
> ID#0] Beginning stream session with /10.128.1.2
> INFO  [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545
> StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac
> ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0
> files(0.000KiB)
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,705
> StreamResultFuture.java:187 - [Stream
> #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] Session with /10.128.1.2 is
> complete
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,707
> StreamResultFuture.java:219 - [Stream
> #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] All sessions completed




On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir  wrote:

> Cassandra version is 3.10, and yes its not a production server. i have
> seen some warnings in logs saying token exist on both servers. other than
> that nothing. if you need any more settings/details please ask. thank you
> for your time
>
>
> On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
>> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:
>>
>>> Cassandra ensure that adding or removing nodes are very easy and that
>>> load is balanced between nodes when a change is made. but it's not working
>>> in my case.
>>> I have a single node C* deployment (with 270 GB of data) and want to
>>> load balance the data on multiple nodes,
>>>
>>
>> I guess it's fair to assume this is not a production "cluster"?
>>
>>
>>> I followed this guide
>>> 
>>>
>>> `nodetool status` shows 2 nodes but load is not balanced between them
>>>
>>> Datacenter: dc1
>>> ===
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  Address  Load   Tokens   Owns (effective)  Host IDRack
>>> UN  10.128.0.7   270.75 GiB  256  48.6%
>>> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
>>> UN  10.128.0.14  414.36 KiB  256  51.4%
>>> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>>>
>>> I can imagine you are referring to Load column here, which shows only
>> 400 KB for the new node.  Did the newly added node actually bootstrap?  Are
>> there any error/warning messages in the Cassandra log?
>>
>>
>>> I also ran 'nodetool repair' on new node but result is same. any
>>> pointers would be appreciated :)
>>>
>>
>> Hm, this is not expected.  Even if the node didn't bootstrap, repair
>> should have streamed the data it is responsible for.
>>
>> conf file of new node
>>>
>>> cluster_name: 'cluster1'
>>>  - seeds: "10.128.0.7"
>>> num_tokens: 256
>>> endpoint_snitch: GossipingPropertyFileSnitch
>>>
>>> I see that you've tried to add the new node as a normal (non-seed) node,
>> which supposed to work and it should have bootstrapped itself.  Which
>> version of Cassandra is it exactly?
>>
>> --
>> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
>> 127-59-707 <+49%20176%2012759707>
>>
>>
>

Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

Cassandra version is 3.10, and yes its not a production server. i have seen
some warnings in logs saying token exist on both servers. other than that
nothing. if you need any more settings/details please ask. thank you for
your time


On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:
>
>> Cassandra ensure that adding or removing nodes are very easy and that
>> load is balanced between nodes when a change is made. but it's not working
>> in my case.
>> I have a single node C* deployment (with 270 GB of data) and want to load
>> balance the data on multiple nodes,
>>
>
> I guess it's fair to assume this is not a production "cluster"?
>
>
>> I followed this guide
>> 
>>
>> `nodetool status` shows 2 nodes but load is not balanced between them
>>
>> Datacenter: dc1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens   Owns (effective)  Host IDRack
>> UN  10.128.0.7   270.75 GiB  256  48.6%
>> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
>> UN  10.128.0.14  414.36 KiB  256  51.4%
>> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>>
>> I can imagine you are referring to Load column here, which shows only 400
> KB for the new node.  Did the newly added node actually bootstrap?  Are
> there any error/warning messages in the Cassandra log?
>
>
>> I also ran 'nodetool repair' on new node but result is same. any pointers
>> would be appreciated :)
>>
>
> Hm, this is not expected.  Even if the node didn't bootstrap, repair
> should have streamed the data it is responsible for.
>
> conf file of new node
>>
>> cluster_name: 'cluster1'
>>  - seeds: "10.128.0.7"
>> num_tokens: 256
>> endpoint_snitch: GossipingPropertyFileSnitch
>>
>> I see that you've tried to add the new node as a normal (non-seed) node,
> which supposed to work and it should have bootstrapped itself.  Which
> version of Cassandra is it exactly?
>
> --
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707 <+49%20176%2012759707>
>
>

Re: Convert single node C* to cluster (rebalancing problem)

[Oleksandr Shulgin]

On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:

> Cassandra ensure that adding or removing nodes are very easy and that load
> is balanced between nodes when a change is made. but it's not working in my
> case.
> I have a single node C* deployment (with 270 GB of data) and want to load
> balance the data on multiple nodes,
>

I guess it's fair to assume this is not a production "cluster"?


> I followed this guide
> 
>
> `nodetool status` shows 2 nodes but load is not balanced between them
>
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens   Owns (effective)  Host IDRack
> UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
> UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>
> I can imagine you are referring to Load column here, which shows only 400
KB for the new node.  Did the newly added node actually bootstrap?  Are
there any error/warning messages in the Cassandra log?


> I also ran 'nodetool repair' on new node but result is same. any pointers
> would be appreciated :)
>

Hm, this is not expected.  Even if the node didn't bootstrap, repair should
have streamed the data it is responsible for.

conf file of new node
>
> cluster_name: 'cluster1'
>  - seeds: "10.128.0.7"
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
>
> I see that you've tried to add the new node as a normal (non-seed) node,
which supposed to work and it should have bootstrapped itself.  Which
version of Cassandra is it exactly?

-- 
Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
127-59-707