OP, I was just looking at your original numbers and I have some questions: 270GB on one node and 414KB on the other, but something close to 50/50 on "Owns(effective)". What replication factor are your keyspaces set up with? 1x or 2x or ??
I would say you are seeing 50/50 because the tokens are allocated 50/50(others on the list please correct what are for me really just assumptions!!!!), but I would hazard a guess that your replication factor is still 1x, so it isn't moving anything around. Or your keyspace rplication is incorrect and isn't being distributed(I have had issues with the AWSMultiRegionSnitch and not getting the region correct[us-east vs us-east-1). It doesn't throw an error, but it doesn't work very well either =) Can you do a 'describe keyspace XXX' and show the first line(the CREATE KEYSPACE line). Mind you, these are all just shots in the dark from here. Cheers, On Tue, Jun 13, 2017 at 3:13 AM Junaid Nasir <jna...@an10.io> wrote: > Is the OP expecting a perfect 50%/50% split? > > > best result I got was 240gb/30gb split, which I think is not properly > balanced. > > >> Also, what are your outputs when you call out specific keyspaces? Do the >> numbers get more even? > > > i don't know what you mean by *call out specific key spaces?* can you > please explain that a bit. > > > If your schema is not modelled correctly you can easily end up unevenly >> distributed data. > > > I think that is the problem. initial 270gb data might not by modeled > correctly. I have run a lot of tests on 270gb data including downsizing it > to 5gb, they all resulted in same uneven distribution. I also tested a > dummy dataset of 2gb which was balanced evenly. coming from rdb, I didn't > give much thought to data modeling. can anyone please point me to some > resources regarding this problem. > > On Tue, Jun 13, 2017 at 3:24 AM, Akhil Mehra <akhilme...@gmail.com> wrote: > >> Great point John. >> >> The OP should also note that data distribution also depends on your >> schema and incoming data profile. >> >> If your schema is not modelled correctly you can easily end up unevenly >> distributed data. >> >> Cheers, >> Akhil >> >> On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthug...@gmail.com> >> wrote: >> >>> Is the OP expecting a perfect 50%/50% split? That, to my experience, is >>> not going to happen, it is almost always shifted from a fraction of a >>> percent to a couple percent. >>> >>> Datacenter: eu-west >>> =================== >>> Status=Up/Down >>> |/ State=Normal/Leaving/Joining/Moving >>> -- Address Load Tokens Owns (effective) Host ID >>> Rack >>> UN XX.XX.XX.XX 22.71 GiB 256 47.6% >>> 57dafdde-2f62-467c-a8ff-c91e712f89c9 1c >>> UN XX.XX.XX.XX 17.17 GiB 256 51.3% >>> d2a65c51-087d-48de-ae1f-a41142eb148d 1b >>> UN XX.XX.XX.XX 26.15 GiB 256 52.4% >>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c 1c >>> UN XX.XX.XX.XX 16.64 GiB 256 50.2% >>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e 1a >>> UN XX.XX.XX.XX 24.39 GiB 256 49.8% >>> fd92525d-edf2-4974-8bc5-a350a8831dfa 1a >>> UN XX.XX.XX.XX 23.8 GiB 256 48.7% >>> bdc597c0-718c-4ef6-b3ef-7785110a9923 1b >>> >>> Though maybe part of what you are experiencing can be cleared up by >>> repair/compaction/cleanup. Also, what are your outputs when you call out >>> specific keyspaces? Do the numbers get more even? >>> >>> Cheers, >>> >>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com> >>> wrote: >>> >>>> auto_bootstrap is true by default. Ensure its set to true. On startup >>>> look at your logs for your auto_bootstrap value. Look at the node >>>> configuration line in your log file. >>>> >>>> Akhil >>>> >>>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote: >>>> >>>>> No, I didn't set it (left it at default value) >>>>> >>>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> wrote: >>>>> >>>>>> Did you make sure auto_bootstrap property is indeed set to [true] >>>>>> when you added the node? >>>>>> >>>>>> >>>>>> >>>>>> *From:* Junaid Nasir [mailto:jna...@an10.io] >>>>>> *Sent:* Monday, June 05, 2017 6:29 AM >>>>>> *To:* Akhil Mehra <akhilme...@gmail.com> >>>>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>; >>>>>> user@cassandra.apache.org >>>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing >>>>>> problem) >>>>>> >>>>>> >>>>>> >>>>>> not evenly, i have setup a new cluster with subset of data (around >>>>>> 5gb). using the configuration above I am getting these results >>>>>> >>>>>> >>>>>> >>>>>> Datacenter: datacenter1 >>>>>> >>>>>> ======================= >>>>>> >>>>>> Status=Up/Down >>>>>> >>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>> >>>>>> -- Address Load Tokens Owns (effective) Host ID >>>>>> Rack >>>>>> >>>>>> UN 10.128.2.1 4.86 GiB 256 44.9% >>>>>> e4427611-c247-42ee-9404-371e177f5f17 rack1 >>>>>> >>>>>> UN 10.128.2.10 725.03 MiB 256 55.1% >>>>>> 690d5620-99d3-4ae3-aebe-8f33af54a08b rack1 >>>>>> >>>>>> is there anything else I can tweak/check to make the distribution >>>>>> even? >>>>>> >>>>>> >>>>>> >>>>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> So now the data is evenly balanced in both nodes? >>>>>> >>>>>> >>>>>> >>>>>> Refer to the following documentation to get a better understanding of >>>>>> the roc_address and the broadcast_rpc_address >>>>>> https://www.instaclustr.com/demystifying-cassandras-broadcast_address/ >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI&e=>. >>>>>> I am surprised that your node started up with rpc_broadcast_address >>>>>> set as this is an unsupported property. I am assuming you are using >>>>>> Cassandra version 3.10. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Akhil >>>>>> >>>>>> >>>>>> >>>>>> On 2/06/2017, at 11:06 PM, Junaid Nasir <jna...@an10.io> wrote: >>>>>> >>>>>> >>>>>> >>>>>> I am able to get it working. I added a new node with following changes >>>>>> >>>>>> #rpc_address:0.0.0.0 >>>>>> >>>>>> rpc_address: 10.128.1.11 >>>>>> >>>>>> #rpc_broadcast_address:10.128.1.11 >>>>>> >>>>>> rpc_address was set to 0.0.0.0, (I ran into a problem previously >>>>>> regarding remote connection and made these changes >>>>>> https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_12236898_apache-2Dcassandra-2Dremote-2Daccess&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=oj8BCLiyBDqqVQNqfGohFeujtqlzAkd-uwS878d4mg4&e=> >>>>>> ) >>>>>> >>>>>> >>>>>> >>>>>> should it be happening? >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin < >>>>>> vla...@winguzone.com> wrote: >>>>>> >>>>>> Did you run "nodetool cleanup" on first node after second was >>>>>> bootstrapped? It should clean rows not belonging to node after tokens >>>>>> changed. >>>>>> >>>>>> >>>>>> >>>>>> Best regards, Vladimir Yudovin, >>>>>> >>>>>> *Winguzone >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com_-3Ffrom-3Dlist&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=Q1M5YRAsw0iUQKOIulEmO72RhdENQCRhpqZSjgxxHos&e=> >>>>>> - Cloud Cassandra Hosting* >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ---- On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir >>>>>> <jna...@an10.io <jna...@an10.io>>* wrote ---- >>>>>> >>>>>> >>>>>> >>>>>> Cassandra ensure that adding or removing nodes are very easy and that >>>>>> load is balanced between nodes when a change is made. but it's not >>>>>> working >>>>>> in my case. >>>>>> >>>>>> I have a single node C* deployment (with 270 GB of data) and want to >>>>>> load balance the data on multiple nodes, I followed this guide >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_cassandra_2.1_cassandra_operations_ops-5Fadd-5Fnode-5Fto-5Fcluster-5Ft.html&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=xnLuX4qqSZxVuY6Gz2NXRdc6TG8J7at9kDhkhBnWpnM&e=> >>>>>> >>>>>> >>>>>> `nodetool status` shows 2 nodes but load is not balanced between them >>>>>> >>>>>> Datacenter: dc1 >>>>>> >>>>>> =============== >>>>>> >>>>>> Status=Up/Down >>>>>> >>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>> >>>>>> -- Address Load Tokens Owns (effective) Host ID >>>>>> Rack >>>>>> >>>>>> UN 10.128.0.7 270.75 GiB 256 48.6% >>>>>> 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1 >>>>>> >>>>>> UN 10.128.0.14 414.36 KiB 256 51.4% >>>>>> 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1 >>>>>> >>>>>> I also ran 'nodetool repair' on new node but result is same. any >>>>>> pointers would be appreciated :) >>>>>> >>>>>> >>>>>> >>>>>> conf file of new node >>>>>> >>>>>> cluster_name: 'cluster1' >>>>>> >>>>>> - seeds: "10.128.0.7" >>>>>> num_tokens: 256 >>>>>> >>>>>> endpoint_snitch: GossipingPropertyFileSnitch >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Junaid >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >> >
