Re: Unbalanced cluster

Awesome utility Avi! Thanks for sharing. On Tue, Jul 11, 2017 at 10:57 AM Avi Kivity wrote: > There is now a readme with some examples and a build file. > > On 07/11/2017 11:53 AM, Avi Kivity wrote: > > Yeah, posting a github link carries an implied undertaking to write a >

Re: Unbalanced cluster

There is now a readme with some examples and a build file. On 07/11/2017 11:53 AM, Avi Kivity wrote: Yeah, posting a github link carries an implied undertaking to write a README file and make it easily buildable. I'll see what I can do. On 07/11/2017 06:25 AM, Nate McCall wrote: You

Re: Unbalanced cluster

Yeah, posting a github link carries an implied undertaking to write a README file and make it easily buildable. I'll see what I can do. On 07/11/2017 06:25 AM, Nate McCall wrote: You wouldnt have a build file laying around for that, would you? On Tue, Jul 11, 2017 at 3:23 PM, Nate McCall

Re: Unbalanced cluster

s = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; The payload column is almost the same size in each record. I understand that an unbalanced cluster may be the result of a bad Primary key, which I believe isn't the case here. Any clue o

Re: Unbalanced cluster

t;> 'false', 'class': >> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} >> AND compression = {'sstable_compression': >> 'org.apache.cassandra.io.compress.LZ4Compressor'} >> AND dclocal_read_repair_chance = 0.1 >> AND default_time

Re: Unbalanced cluster

You wouldnt have a build file laying around for that, would you? On Tue, Jul 11, 2017 at 3:23 PM, Nate McCall wrote: > On Tue, Jul 11, 2017 at 3:20 AM, Avi Kivity wrote: > >> >> >> >> [1] https://github.com/avikivity/shardsim >> > > Avi, that's super

Re: Unbalanced cluster

On Tue, Jul 11, 2017 at 3:20 AM, Avi Kivity wrote: > > > > [1] https://github.com/avikivity/shardsim > Avi, that's super handy - thanks for posting.

Re: Unbalanced cluster

the reason for the default of 256 vnodes is because at that many tokens the random distribution of tokens is enough to balance out each nodes token allocation almost evenly. any less and some nodes will get far more unbalanced, as Avi has shown. In 3.0 there is a new token allocating algorithm

Re: Unbalanced cluster

nds = 172000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; The payload column is almost the same size in each record. I understand that an unbalanced cl

Unbalanced cluster

ve = 0 AND gc_grace_seconds = 172000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; The payload column is almost the same size in each record. I understand that an unbal

Re: Totally unbalanced cluster

Hi, Regarding the documentation I've already knew: - thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html (From myself, how to handle tombstones) - http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html

Re: Totally unbalanced cluster

Hi, > but it's so easy to add nodes Apache Cassandra has some kind of magic pieces ;-). Sometimes it is dark magic though :p. Yet adding a node is indeed not harder when using NetworkTopologyStrategy, as Jon mentioned above, once the configuration is done once properly. > Number of keys

Re: Totally unbalanced cluster

Adding nodes with NTS is easier, in my opinion. You don’t need to worry about replica placement, if you do it right. > On May 4, 2017, at 7:43 AM, Cogumelos Maravilha > wrote: > > Hi Alain thanks for your kick reply. > > > Regarding SimpleStrategy perhaps you

Re: Totally unbalanced cluster

Hi Alain thanks for your kick reply. Regarding SimpleStrategy perhaps you are right but it's so easy to add nodes. I'm *not* using vnodes and the default 256. The information that I've posted it a regular nodetool status keyspace. My partition key is a sequencial big int but nodetool cfstatus

Re: Totally unbalanced cluster

Hi, CREATE KEYSPACE mykeyspace WITH replication = {'class': > 'SimpleStrategy', 'replication_factor': '2'} AND durable_writes = false; The SimpleStrategy is never recommended for production clusters as it does not recognise racks or datacenter, inducing possible availability issues and

Totally unbalanced cluster

Hi all, I'm using C* 3.10. CREATE KEYSPACE mykeyspace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '2'} AND durable_writes = false; CREATE TABLE mykeyspace.data ( id bigint PRIMARY KEY, kafka text ) WITH bloom_filter_fp_chance = 0.5 AND caching = {'keys':

Re: Property file snitch and unbalanced cluster

Hi Robert, What has happened is that you have now two datacenters in your cluster. The way they replicate information will depend on your keyspace settings. Regarding your process I don't think it is safe to do it that way. I'd start off by decommissioning nodes 4 and 5 so that your cluster is

Property file snitch and unbalanced cluster

Hi guys, I have a cluster with 5 nodes, cassandra 3.0.5, RF=2, PropertyFileSnitch. Three of the nodes were added initially, DC=PRD. The other two were added more recently. The default for unknown nodes default=DC1:r1 (which is bad in my case, as it would have been nice to have PRD as default)

Re: Unbalanced cluster with RandomPartitioner

Setting a token outside of the partitioner range sounds like a bug. It's mostly an issue with the RP, but I guess a custom partitioner may also want to validate tokens are within a range. Can you report it to https://issues.apache.org/jira/browse/CASSANDRA Thanks - Aaron

Re: Unbalanced cluster with RandomPartitioner

I thought about our issue again and was thinking, maybe the describeOwnership should take into account, if a token is outside the partitioners maximum token range? To recap our problem: we had tokens, that were apart by 12.5% of the token range 2**127, however, we had an offset on each token,

Re: Unbalanced cluster with RandomPartitioner

On 19.01.2012, at 20:15, Narendra Sharma wrote: I believe you need to move the nodes on the ring. What was the load on the nodes before you added 5 new nodes? Its just that you are getting data in certain token range more than others. With three nodes, it was also imbalanced. What I don't

Re: Unbalanced cluster with RandomPartitioner

Thanks for all the responses! I found our problem: Using the Random Partitioner, the key range is from 0..2**127.When we added nodes, we generated the keys and out of convenience, we added an offset to the tokens because the move was easier like that. However, we did not execute the modulo

Re: Unbalanced cluster with RandomPartitioner

On 18.01.2012, at 02:19, Maki Watanabe wrote: Are there any significant difference of number of sstables on each nodes? No, no significant difference there. Actually, node 8 is among those with more sstables but with the least load (20GB) On 17.01.2012, at 20:14, Jeremiah Jordan wrote: Are you

Re: Unbalanced cluster with RandomPartitioner

2012/1/19 aaron morton aa...@thelastpickle.com: If you have performed any token moves the data will not be deleted until you run nodetool cleanup. We did that after adding nodes to the cluster. And then, the cluster wasn't balanced either. Also, does the Load really account for dead data, or is

Re: Unbalanced cluster with RandomPartitioner

Load reported from node tool ring is the live load, which means SSTables that the server has open and will read from during a request. This will include tombstones, expired and over written data. nodetool ctstats also includes dead load, which is sstables that are in use but still on disk.

Re: Unbalanced cluster with RandomPartitioner

I believe you need to move the nodes on the ring. What was the load on the nodes before you added 5 new nodes? Its just that you are getting data in certain token range more than others. -Naren On Thu, Jan 19, 2012 at 3:22 AM, Marcel Steinbach marcel.steinb...@chors.de wrote: On 18.01.2012,

Re: Unbalanced cluster with RandomPartitioner

If you have performed any token moves the data will not be deleted until you run nodetool cleanup. To get a baseline I would run nodetool compact to do major compaction and purge any tomb stones as others have said. Cheers - Aaron Morton Freelance Developer @aaronmorton

Unbalanced cluster with RandomPartitioner

Hi, we're using RP and have each node assigned the same amount of the token space. The cluster looks like that: Address Status State LoadOwnsToken

Re: Unbalanced cluster with RandomPartitioner

Have you tried running repair first on each node? Also, verify using df -h on the data dirs On Tue, Jan 17, 2012 at 7:34 AM, Marcel Steinbach marcel.steinb...@chors.de wrote: Hi, we're using RP and have each node assigned the same amount of the token space. The cluster looks like that:

Re: Unbalanced cluster with RandomPartitioner

We are running regular repairs, so I don't think that's the problem. And the data dir sizes match approx. the load from the nodetool. Thanks for the advise, though. Our keys are digits only, and all contain a few zeros at the same offsets. I'm not that familiar with the md5 algorithm, but I

Re: Unbalanced cluster with RandomPartitioner

Are you deleting data or using TTL's? Expired/deleted data won't go away until the sstable holding it is compacted. So if compaction has happened on some nodes, but not on others, you will see this. The disparity is pretty big 400Gb to 20GB, so this probably isn't the issue, but with our

Re: Unbalanced cluster with RandomPartitioner

Are there any significant difference of number of sstables on each nodes? 2012/1/18 Marcel Steinbach marcel.steinb...@chors.de: We are running regular repairs, so I don't think that's the problem. And the data dir sizes match approx. the load from the nodetool. Thanks for the advise, though.

Re: how to solve one node is in heavy load in unbalanced cluster

-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: how to solve one node is in heavy load in unbalanced cluster

move first and reair later. You are moving some nodes data as is so it's no worse than what you have right now. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317

Re: how to solve one node is in heavy load in unbalanced cluster

this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: how to solve one node is in heavy load in unbalanced cluster

are moving some nodes data as is so it's no worse than what you have right now. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html Sent from the cassandra-u

Re: how to solve one node is in heavy load in unbalanced cluster

right now. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: how to solve one node is in heavy load in unbalanced cluster

any help? thanks! On Fri, Jul 29, 2011 at 12:05 PM, Yan Chunlu springri...@gmail.com wrote: and by the way, my RF=3 and the other two nodes have much more capacity, why does they always routed the request to node3? coud I do a rebalance now? before node repair? On Fri, Jul 29, 2011 at

Re: how to solve one node is in heavy load in unbalanced cluster

.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6638649.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: how to solve one node is in heavy load in unbalanced cluster

-in-unbalanced-cluster-tp6630827p6638649.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: how to solve one node is in heavy load in unbalanced cluster

.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6638649.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: how to solve one node is in heavy load in unbalanced cluster

are moving some nodes data as is so it's no worse than what you have right now. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html Sent from the cassandra-u

Re: how to solve one node is in heavy load in unbalanced cluster

that token to perform move. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6638649.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: how to solve one node is in heavy load in unbalanced cluster

-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

how to solve one node is in heavy load in unbalanced cluster

I have three nodes and RF=3.here is the current ring: Address Status State Load Owns Token 84944475733633104818662955375549269696 node1 Up Normal 15.32 GB 81.09% 52773518586096316348543097376923124102 node2 Up Normal 22.51 GB 10.48% 70597222385644499881390884416714081360 node3 Up Normal 56.1 GB

Re: how to solve one node is in heavy load in unbalanced cluster

Dropped read message might be an indicator of capacity issue. We experienced the similar issue with 0.7.6. We ended up adding two extra nodes and physically rebooted the offending node(s). The entire cluster then calmed down. On Thu, Jul 28, 2011 at 2:24 PM, Yan Chunlu springri...@gmail.com

Re: how to solve one node is in heavy load in unbalanced cluster

add new nodes seems added more pressure to the cluster? how about your data size? On Fri, Jul 29, 2011 at 4:16 AM, Frank Duan fr...@aimatch.com wrote: Dropped read message might be an indicator of capacity issue. We experienced the similar issue with 0.7.6. We ended up adding two extra

Re: how to solve one node is in heavy load in unbalanced cluster

and by the way, my RF=3 and the other two nodes have much more capacity, why does they always routed the request to node3? coud I do a rebalance now? before node repair? On Fri, Jul 29, 2011 at 12:01 PM, Yan Chunlu springri...@gmail.com wrote: add new nodes seems added more pressure to the

fixing unbalanced cluster !?

I got myself into a situation where one node (10.47.108.100) has a lot more data than the other nodes. In fact, the 1 TB disk on this node is almost full. I added 3 new nodes and let cassandra automatically calculate new tokens by taking the highest loaded nodes. Unfortunately there is

Re: fixing unbalanced cluster !?

Because you were able to successfully run repair you can follow up with a nodetool cleanup which will git rid of some of the extraneous data on that (bigger) node. You're also assured after you run repair that entropy beteen the nodes is minimal. Assuming you're using the random ordered

Re: fixing unbalanced cluster !?

Thanks Ben. That's what I was afraid I had to do. I can see how it's a lot easier if you simply double the cluster when adding capacity. Jon On Jun 9, 2011, at 4:44 PM, Benjamin Coverston wrote: Because you were able to successfully run repair you can follow up with a nodetool cleanup