subject:"Totally unbalanced cluster"

Re: Totally unbalanced cluster

2017-05-05 Thread Cogumelos Maravilha

Hi,

Regarding the documentation I've already knew:

- thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html

(From myself, how to handle tombstones)
- http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html (From
Alexander, a coworker @TLP - TWCS and expiring tables)

Anyway fantastic docs.

I desperately need to free up disk space. nodetool repair can do an
anticompaction.
In my case C* is only used to insert data that expires with a TTL of 18
days. No updates, or deletes. Some selects using the partition key.
gc_grace is defined to 3 hours.

Best practices do free up disk space please?

Thanks in advance



On 05/05/2017 03:09 PM, Alain RODRIGUEZ wrote:
> Hi,
>  
>
> but it's so easy to add nodes
>
>
> Apache Cassandra has some kind of magic pieces ;-). Sometimes it is
> dark magic though :p. Yet adding a node is indeed not harder when
> using NetworkTopologyStrategy, as Jon mentioned above, once the
> configuration is done once properly.
>
> Number of keys (estimate): 442779640
>
> Number of keys (estimate): 736380940
>
> Number of keys (estimate): 451097313
>
>
> This is indeed possibly, and most certainly creating imbalances. But
> also look at the partition size when using 'nodetool cfstats', using
> the previous information and the 'Compacted partition mean bytes', you
> should have an idea how much the disk space used is imbalanced. If you
> would like more details on the partition size distribution, partition
> size percentile are available using 'nodetool cfhistograms'.
>
> Regarding the global load (CPU, GC, disk IO, etc), it also depends on
> the workload (ie, what partitions are being read).
>
> *Should I use nodetool rebuild?*
>
>
> No, I see no reason. This command, 'nodetool rebuild' is meant to be
> used when adding a new datacenter to the cluster. Which, by the way,
> will not happen as long as you are using the 'SimpleStrategy', that
> basically creates one big ring and consider all the nodes as being
> part of it, no matter their placement in the network, if I remember
> correctly.
>
> The nodetool repair by default in this C* version is incremental
> and since the repair is run in all nodes in different hours 
>
>
> Incremental repairs are quite new to me. But I heard they bring some
> issues, often due to anti-compactions inducing a high number of
> SSTables and a growing number of compactions pending. But it does not
> look bad in your case. 
>
> Yet the '-pr' option should not be used when doing incremental
> repairs. This thread mentions it and is probably worth
> reading: https://groups.google.com/forum/#!topic/nosql-databases/peTArLfhXMU
> .
> Also I believe it is mentioned in the video about repairs from
> Alexander I shared in my last mail.
>
> and I don't want snapshots that's why I'm cleaning twice a day
> (not sure that with -pr a snapshot is created).
>
>
> So the option using snapshots is not '-pr', but '-seq' (sequential) or
> '-par' (parallel), more
> info: 
> https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html
>
> If you want to keep using sequential repairs, then you could check the
> snpashots automaticly generated names, and aim at deleting them
> specifically to prevent you from removing an other manually created
> and possibly important snapshot.
>
> I'm using garbagecollect to force the cleanup since I'm running
> out of space.
>
>
> Oh that's a whole topic. These blogposts should hopefully be helpful:
>
> - thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
> 
> (From myself, how to handle tombstones)
> - http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html (From
> Alexander, a coworker @TLP - TWCS and expiring tables)
>
> Hopefully some information picked from those 2 blog posts might help
> you freeing some disk space.
>
> It is probably not needed to use 'garbagecollect' as a routine
> operation. Some tuning in the compaction strategy or options (using
> defaults currently) might be enough to solve the issue.
>
> Yet the data is not correctly distributed. Something in the data model
> design is inducing it. The primary key (hashed) is what is used to
> affect the data to a specific node. Also a variable partition size can
> also lead to hotspots.
>
> As a side note, I strongly believe that understanding internals is
> very important to operate Apache Cassandra correctly, I mean playing
> with it to learn can put you in some undesirable situations. That's
> why I keep mentioning some blog posts, talks or documentations that I
> think could be helpful to know Apache Cassandra internals and
> processes a bit more.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
>

Re: Totally unbalanced cluster

2017-05-05 Thread Alain RODRIGUEZ

Hi,

> but it's so easy to add nodes

Apache Cassandra has some kind of magic pieces ;-). Sometimes it is dark
magic though :p. Yet adding a node is indeed not harder when using
NetworkTopologyStrategy, as Jon mentioned above, once the configuration is
done once properly.

> Number of keys (estimate): 442779640
>
> Number of keys (estimate): 736380940
>
> Number of keys (estimate): 451097313
>

This is indeed possibly, and most certainly creating imbalances. But also
look at the partition size when using 'nodetool cfstats', using the
previous information and the 'Compacted partition mean bytes', you should
have an idea how much the disk space used is imbalanced. If you would like
more details on the partition size distribution, partition size percentile
are available using 'nodetool cfhistograms'.

Regarding the global load (CPU, GC, disk IO, etc), it also depends on the
workload (ie, what partitions are being read).

*Should I use nodetool rebuild?*

No, I see no reason. This command, 'nodetool rebuild' is meant to be used
when adding a new datacenter to the cluster. Which, by the way, will not
happen as long as you are using the 'SimpleStrategy', that basically
creates one big ring and consider all the nodes as being part of it, no
matter their placement in the network, if I remember correctly.

The nodetool repair by default in this C* version is incremental and since
> the repair is run in all nodes in different hours

Incremental repairs are quite new to me. But I heard they bring some
issues, often due to anti-compactions inducing a high number of SSTables
and a growing number of compactions pending. But it does not look bad in
your case.

Yet the '-pr' option should not be used when doing incremental repairs.
This thread mentions it and is probably worth reading:
https://groups.google.com/forum/#!topic/nosql-databases/peTArLfhXMU. Also I
believe it is mentioned in the video about repairs from Alexander I shared
in my last mail.

and I don't want snapshots that's why I'm cleaning twice a day (not sure
> that with -pr a snapshot is created).

So the option using snapshots is not '-pr', but '-seq' (sequential) or
'-par' (parallel), more info:
https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html

If you want to keep using sequential repairs, then you could check the
snpashots automaticly generated names, and aim at deleting them
specifically to prevent you from removing an other manually created and
possibly important snapshot.

I'm using garbagecollect to force the cleanup since I'm running out of
> space.

Oh that's a whole topic. These blogposts should hopefully be helpful:

- thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html (From
myself, how to handle tombstones)
- http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html (From Alexander,
a coworker @TLP - TWCS and expiring tables)

Hopefully some information picked from those 2 blog posts might help you
freeing some disk space.

It is probably not needed to use 'garbagecollect' as a routine operation.
Some tuning in the compaction strategy or options (using defaults
currently) might be enough to solve the issue.

Yet the data is not correctly distributed. Something in the data model
design is inducing it. The primary key (hashed) is what is used to affect
the data to a specific node. Also a variable partition size can also lead
to hotspots.

As a side note, I strongly believe that understanding internals is very
important to operate Apache Cassandra correctly, I mean playing with it to
learn can put you in some undesirable situations. That's why I keep
mentioning some blog posts, talks or documentations that I think could be
helpful to know Apache Cassandra internals and processes a bit more.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2017-05-04 17:10 GMT+01:00 Jon Haddad :

> Adding nodes with NTS is easier, in my opinion.  You don’t need to worry
> about replica placement, if you do it right.
>
> On May 4, 2017, at 7:43 AM, Cogumelos Maravilha <
> cogumelosmaravi...@sapo.pt> wrote:
>
> Hi Alain thanks for your kick reply.
>
>
> Regarding SimpleStrategy perhaps you are right but it's so easy to add
> nodes.
>
> I'm *not* using vnodes and the default 256. The information that I've
> posted it a regular nodetool status keyspace.
>
> My partition key is a sequencial big int but nodetool cfstatus shows that
> the number of keys are not balanced (data from 3 nodes):
>
> Number of keys (estimate): 442779640
>
> Number of keys (estimate): 736380940
>
> Number of keys (estimate): 451097313
>
> *Should I use nodetool rebuild?*
>
> Running:
>
> nodetool getendpoints mykeyspace data 9213395123941039285
>
> 10.1.1.52
> 10.1.1.185
>
> nodetool getendpoints mykeyspace data 9213395123941039286
>
> 10.1.1.161
> 10.1.1.19
>
> All nodes are working hard because my

Re: Totally unbalanced cluster

2017-05-04 Thread Jon Haddad

Adding nodes with NTS is easier, in my opinion.  You don’t need to worry about 
replica placement, if you do it right.

> On May 4, 2017, at 7:43 AM, Cogumelos Maravilha  
> wrote:
> 
> Hi Alain thanks for your kick reply.
> 
> 
> Regarding SimpleStrategy perhaps you are right but it's so easy to add nodes.
> 
> I'm not using vnodes and the default 256. The information that I've posted it 
> a regular nodetool status keyspace.
> 
> My partition key is a sequencial big int but nodetool cfstatus shows that the 
> number of keys are not balanced (data from 3 nodes):
> 
> Number of keys (estimate): 442779640
> 
> Number of keys (estimate): 736380940
> 
> Number of keys (estimate): 451097313
> 
> Should I use nodetool rebuild?
> 
> Running:
> 
> nodetool getendpoints mykeyspace data 9213395123941039285
> 
> 10.1.1.52
> 10.1.1.185
> 
> nodetool getendpoints mykeyspace data 9213395123941039286
> 
> 10.1.1.161
> 10.1.1.19
> All nodes are working hard because my TTL is for 18 days and daily data 
> ingestion is around 120,000,000 records:
> nodetool compactionstats -H
> pending tasks: 3
> - mykeyspace.data: 3
> 
> id   compaction type keyspace  table 
> completed  total  unit  progress
> c49599b1-308d-11e7-ba5b-67e232f1bee1 Remove deleted data mykeyspace data 
> 133.89 GiB 158.33 GiB bytes 84.56%  
> c49599b0-308d-11e7-ba5b-67e232f1bee1 Remove deleted data mykeyspace data 
> 136.2 GiB  278.96 GiB bytes 48.83%
> 
> Active compaction remaining time :   0h00m00s
> 
> 
> nodetool compactionstats -H
> pending tasks: 2
> - mykeyspace.data: 2
> 
> id   compaction type keyspace  table 
> completed total  unit  progress
> b6e8ce80-30d4-11e7-a2be-9b830f114108 Compaction  mykeyspace data 4.05 GiB 
>  133.02 GiB bytes 3.04%   
> Active compaction remaining time :   2h17m34s
> 
> The nodetool repair by default in this C* version is incremental and since 
> the repair is run in all nodes in different hours and I don't want snapshots 
> that's why I'm cleaning twice a day (not sure that with -pr a snapshot is 
> created).
> 
> The cleanup was already remove was there because last node was created a few 
> days ago.
> 
> I'm using garbagecollect to force the cleanup since I'm running out of space.
> 
> 
> Regards.
> 
> 
> 
> On 05/04/2017 12:50 PM, Alain RODRIGUEZ wrote:
>> Hi,
>> 
>> CREATE KEYSPACE mykeyspace WITH replication = {'class':
>> 'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = false;
>> 
>> The SimpleStrategy is never recommended for production clusters as it does 
>> not recognise racks or datacenter, inducing possible availability issues and 
>> unpredictable latency when using those. I would not even use it for testing 
>> purposes, I see no point in most cases.
>> 
>> Even if this should be changed, carefully but as soon as possible imho, it 
>> is probably not related to your main issue at hand.
>> 
>> If nodes are imbalanced, there are 3 mains questions that come to my mind:
>> 
>> Are the token well distributed among the available nodes?
>> Is the data correctly balanced on the token ring (i.e. are the 'id' values 
>> of 'mykeyspace.data' table well spread between the nodes?
>> Are the compaction processes running smoothly on every nodes
>> 
>> Point 1 depends on whether you are using vnodes or not and what number of 
>> vnodes ('num_token' in cassandra.yaml).
>> If not using vnodes, you have to manually set the positions of the nodes and 
>> move them around when adding more nodes so thing remain balanced
>> If using vnodes, make sure to use a high enough number of vnodes so 
>> distribution is 'good enough' (More than 32 in most cases, default is 256, 
>> which lead to quite balanced rings, but brings other issues)
>> 
>> UN  10.1.1.161  398.39 GiB  256  28.9%
>> UN  10.1.1.19   765.32 GiB  256  29.9%
>> UN  10.1.1.52   574.24 GiB  256  28.2%
>> UN  10.1.1.213  817.56 GiB  256  28.2%
>> UN  10.1.1.85   638.82 GiB  256  28.2%
>> UN  10.1.1.245  408.95 GiB  256  28.7%
>> UN  10.1.1.185  574.63 GiB  256  27.9%
>> 
>> You can have the token ownership information by running 'nodetool status 
>> '. Adding the keyspace name in the command give you the real 
>> ownership. Also, RF = 2 means the total of the ownership should be 200%, 
>> ideally evenly balanced. I am not sure about the command you ran here. Also 
>> as a global advice, let us the command you ran and what you expect us to see 
>> in the output.
>> 
>> Still the tokens seems to be well distributed, and I guess you are using the 
>> default 'num_token': 256. So I believe you are not having this issue. But 
>> the delta between the data hold on each node is up to x2 (400 GB on some 
>> nodes, 800 GB on some others).
>> 
>> Point 2 highly depends on the workload. Are your partitions evenly 
>> distributed among the nodes? It depends on your primary key. Using an UUID

Re: Totally unbalanced cluster

2017-05-04 Thread Cogumelos Maravilha

Hi Alain thanks for your kick reply.


Regarding SimpleStrategy perhaps you are right but it's so easy to add
nodes.

I'm *not* using vnodes and the default 256. The information that I've
posted it a regular nodetool status keyspace.

My partition key is a sequencial big int but nodetool cfstatus shows
that the number of keys are not balanced (data from 3 nodes):

Number of keys (estimate): 442779640

Number of keys (estimate): 736380940

Number of keys (estimate): 451097313

*Should I use nodetool rebuild?*

Running:

nodetool getendpoints mykeyspace data 9213395123941039285

10.1.1.52
10.1.1.185

nodetool getendpoints mykeyspace data 9213395123941039286

10.1.1.161
10.1.1.19

All nodes are working hard because my TTL is for 18 days and daily data
ingestion is around 120,000,000 records:

nodetool compactionstats -H
pending tasks: 3
- mykeyspace.data: 3

id   compaction type keyspace 
table completed  total  unit  progress
c49599b1-308d-11e7-ba5b-67e232f1bee1 Remove deleted data mykeyspace data
133.89 GiB 158.33 GiB bytes 84.56% 
c49599b0-308d-11e7-ba5b-67e232f1bee1 Remove deleted data mykeyspace data
136.2 GiB  278.96 GiB bytes 48.83%

Active compaction remaining time :   0h00m00s


nodetool compactionstats -H
pending tasks: 2
- mykeyspace.data: 2

id   compaction type keyspace  table
completed total  unit  progress
b6e8ce80-30d4-11e7-a2be-9b830f114108 Compaction  mykeyspace data
4.05 GiB  133.02 GiB bytes 3.04%  
Active compaction remaining time :   2h17m34s


The nodetool repair by default in this C* version is incremental and
since the repair is run in all nodes in different hours and I don't want
snapshots that's why I'm cleaning twice a day (not sure that with -pr a
snapshot is created).

The cleanup was already remove was there because last node was created a
few days ago.

I'm using garbagecollect to force the cleanup since I'm running out of
space.


Regards.


On 05/04/2017 12:50 PM, Alain RODRIGUEZ wrote:
> Hi,
>
> CREATE KEYSPACE mykeyspace WITH replication = {'class':
> 'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes =
> false;
>
>
> The SimpleStrategy is never recommended for production clusters as it
> does not recognise racks or datacenter, inducing possible availability
> issues and unpredictable latency when using those. I would not even
> use it for testing purposes, I see no point in most cases.
>
> Even if this should be changed, carefully but as soon as possible
> imho, it is probably not related to your main issue at hand.
>
> If nodes are imbalanced, there are 3 mains questions that come to my mind:
>
>  1. Are the token well distributed among the available nodes?
>  2. Is the data correctly balanced on the token ring (i.e. are the
> 'id' values of 'mykeyspace.data' table well spread between the nodes?
>  3. Are the compaction processes running smoothly on every nodes
>
>
> *Point 1* depends on whether you are using vnodes or not and what
> number of vnodes ('num_token' in cassandra.yaml).
>
>   * If not using vnodes, you have to manually set the positions of the
> nodes and move them around when adding more nodes so thing remain
> balanced
>   * If using vnodes, make sure to use a high enough number of vnodes
> so distribution is 'good enough' (More than 32 in most cases,
> default is 256, which lead to quite balanced rings, but brings
> other issues)
>
>
> UN  10.1.1.161  398.39 GiB  256  28.9%
> UN  10.1.1.19   765.32 GiB  256  29.9%
> UN  10.1.1.52   574.24 GiB  256  28.2%
> UN  10.1.1.213  817.56 GiB  256  28.2%
> UN  10.1.1.85   638.82 GiB  256  28.2%
> UN  10.1.1.245  408.95 GiB  256  28.7%
> UN  10.1.1.185  574.63 GiB  256  27.9%
>
>
> You can have the token ownership information by running 'nodetool
> status '. Adding the keyspace name in the command give you
> the real ownership. Also, RF = 2 means the total of the ownership
> should be 200%, ideally evenly balanced. I am not sure about the
> command you ran here. Also as a global advice, let us the command you
> ran and what you expect us to see in the output.
>
> Still the tokens seems to be well distributed, and I guess you are
> using the default 'num_token': 256. So I believe you are not having
> this issue. But the delta between the data hold on each node is up to
> x2 (400 GB on some nodes, 800 GB on some others).
>
> *Point 2* highly depends on the workload. Are your partitions evenly
> distributed among the nodes? It depends on your primary key. Using an
> UUID as the partition key is often a good idea, but it depends on your
> needs as well, of course. You could look at the distribution on the
> distinct nodes through: 'nodetool cfstats'.
> *
> *
> *Point 3* : even if the tokens are perfectly distributed and the
> primary key perfectly randomized, some node can have some disk issue
> or any

Re: Totally unbalanced cluster

2017-05-04 Thread Alain RODRIGUEZ

Hi,

CREATE KEYSPACE mykeyspace WITH replication = {'class':
> 'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = false;


The SimpleStrategy is never recommended for production clusters as it does
not recognise racks or datacenter, inducing possible availability issues
and unpredictable latency when using those. I would not even use it for
testing purposes, I see no point in most cases.

Even if this should be changed, carefully but as soon as possible imho, it
is probably not related to your main issue at hand.

If nodes are imbalanced, there are 3 mains questions that come to my mind:


   1. Are the token well distributed among the available nodes?
   2. Is the data correctly balanced on the token ring (i.e. are the 'id'
   values of 'mykeyspace.data' table well spread between the nodes?
   3. Are the compaction processes running smoothly on every nodes


*Point 1* depends on whether you are using vnodes or not and what number of
vnodes ('num_token' in cassandra.yaml).

   - If not using vnodes, you have to manually set the positions of the
   nodes and move them around when adding more nodes so thing remain balanced
   - If using vnodes, make sure to use a high enough number of vnodes so
   distribution is 'good enough' (More than 32 in most cases, default is 256,
   which lead to quite balanced rings, but brings other issues)


UN  10.1.1.161  398.39 GiB  256  28.9%
> UN  10.1.1.19   765.32 GiB  256  29.9%
> UN  10.1.1.52   574.24 GiB  256  28.2%
> UN  10.1.1.213  817.56 GiB  256  28.2%
> UN  10.1.1.85   638.82 GiB  256  28.2%
> UN  10.1.1.245  408.95 GiB  256  28.7%
> UN  10.1.1.185  574.63 GiB  256  27.9%


You can have the token ownership information by running 'nodetool status
'. Adding the keyspace name in the command give you the real
ownership. Also, RF = 2 means the total of the ownership should be 200%,
ideally evenly balanced. I am not sure about the command you ran here. Also
as a global advice, let us the command you ran and what you expect us to
see in the output.

Still the tokens seems to be well distributed, and I guess you are using
the default 'num_token': 256. So I believe you are not having this issue.
But the delta between the data hold on each node is up to x2 (400 GB on
some nodes, 800 GB on some others).

*Point 2* highly depends on the workload. Are your partitions evenly
distributed among the nodes? It depends on your primary key. Using an UUID
as the partition key is often a good idea, but it depends on your needs as
well, of course. You could look at the distribution on the distinct nodes
through: 'nodetool cfstats'.

*Point 3* : even if the tokens are perfectly distributed and the primary
key perfectly randomized, some node can have some disk issue or any other
reason having the compactions falling behind. This would lead to this node
to hold more data and note evicting tombstones properly in some cases,
increasing disk space used. Other than that, you can have a big SSTable
being compacted on a node, having the size of the node growing quite
suddenly (that's why 50 to 20% of the disk should always be free, depending
on the compaction strategy in use and the number of concurrent
compactions). Here, running 'nodetool compactionstats -H' on all the nodes
would probably help you to troubleshoot.

*About crontab*


> 08 05   * * *   rootnodetool repair -pr
> 11 11   * * *   rootfstrim -a
> 04 12   * * *   rootnodetool clearsnapshot
> 33 13   * * 2   rootnodetool cleanup
> 35 15   * * *   rootnodetool garbagecollect
> 46 19   * * *   rootnodetool clearsnapshot
> 50 23   * * *   rootnodetool flush
>

I don't understand what you try to achieve with some of the commands:

nodetool repair -pr


Repairing the cluster regularly is good in most cases, but as default
changes with version, I would specify if the repair is supposed to be
'incremental' or 'full', if it is supposed to be 'sequential' or 'parallel'
for example. Also, as the dataset growth, some issue will appear with
repairs.Just search for 'repairs cassandra' on google or any search engine
you are using and you will see that repair is a complex topic. Look for
videos and you will find a lot of informations about it from nice talks
like these 2 from the last summit:

https://www.youtube.com/watch?v=FrF8wQuXXks
https://www.youtube.com/watch?v=1Sz_K8UID6E

Also some nice tools exist to help with repairs:

The Reaper (originally made at Spotify now maintained by The Last Pickle):
https://github.com/thelastpickle/cassandra-reaper
'cassandra_range_repair.py':  https://github.com/
BrianGallew/cassandra_range_repair

11 11   * * *   rootfstrim -a


I am not really sure about this one but it looks good as long as the
'fstrim' do not create performance issue while it is running it seems fine.

04 12   * * *   rootnodetool clearsnapshot


This will automatically erase any snapshot you might want to keep. It might
be good to

Totally unbalanced cluster

2017-05-04 Thread Cogumelos Maravilha

Hi all,

I'm using C* 3.10.

CREATE KEYSPACE mykeyspace WITH replication = {'class':
'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = false;

CREATE TABLE mykeyspace.data (
id bigint PRIMARY KEY,
kafka text
) WITH bloom_filter_fp_chance = 0.5
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '10', 'compaction_window_unit': 'HOURS',
'max_threshold': '32', 'min_threshold': '6'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 0.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 1555200
AND gc_grace_seconds = 10800
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

UN  10.1.1.161  398.39 GiB  256  28.9%
UN  10.1.1.19   765.32 GiB  256  29.9%
UN  10.1.1.52   574.24 GiB  256  28.2%
UN  10.1.1.213  817.56 GiB  256  28.2%
UN  10.1.1.85   638.82 GiB  256  28.2%
UN  10.1.1.245  408.95 GiB  256  28.7%
UN  10.1.1.185  574.63 GiB  256  27.9%

At crontab in all nodes (only changes the time):

08 05   * * *   rootnodetool repair -pr
11 11   * * *   rootfstrim -a
04 12   * * *   rootnodetool clearsnapshot
33 13   * * 2   rootnodetool cleanup
35 15   * * *   rootnodetool garbagecollect
46 19   * * *   rootnodetool clearsnapshot
50 23   * * *   rootnodetool flush

I can I fixed this?

Thanks in advance.



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Totally unbalanced cluster

Re: Totally unbalanced cluster

Re: Totally unbalanced cluster

Re: Totally unbalanced cluster

Re: Totally unbalanced cluster

Totally unbalanced cluster

6 matches

Site Navigation

Mail list logo

Footer information