Re: Metrics matrix: migrate 2.1.x metrics to 2.2.x+

[Alain RODRIGUEZ]

Hello Carl,

I guess we can use bean_regex to do specific targetted metrics for the
> important tables anyway.
>

Yes, this would work, but 350 is very limited for Cassandra dashboards. We
have a LOT of metrics available.

Datadog 350 metric limit is a PITA for tables once you get over 10 tables
>

I noticed this while I was working on providing default dashboards for
Cassandra-Datadog integration. I was told by Datadog team it would not be
an issue for users, that I should not care about it. As you pointed out,
per table metrics quickly increase the total number of metrics we need to
collect.

I believe you can set the following option: *"max_returned_metrics: 1000"* -
it can be used if metrics are missing to increase the limit of the number
of collected metrics. Be aware of CPU utilization that this might imply
(greatly improved in dd-agent version 6+ I believe -thanks Datadog teams
for that- making this fully usable for Cassandra). This option should go in
the *cassandra.yaml* file for Cassandra integrations, off the top of my
head.

Also, do not hesitate to reach to Datadog directly for this kind of
questions, I have always been very happy with their support so far, I am
sure they would guide you through this as well, probably better than we can
do :). It also provides them with feedback on what people are struggling
with I imagine.

I am interested to know if you still have issues getting more metrics
(option above not working / CPU under too much load) as this would make the
dashboards we built mostly unusable for clusters with more tables. We might
then need to review the design.

As a side note, I believe metrics are handled the same way cross version,
they got the same name/label for C*2.1, 2.2 and 3+ on Datadog. There is an
abstraction layer that removes this complexity (if I remember well, we
built those dashboards a while ago).

C*heers
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le lun. 1 oct. 2018 à 19:38, Carl Mueller
 a écrit :

> That's great too, thank you.
>
> Datadog 350 metric limit is a PITA for tables once you get over 10 tables,
> but I guess we can use bean_regex to do specific targetted metrics for the
> important tables anyway.
>
> On Mon, Oct 1, 2018 at 4:21 AM Alain RODRIGUEZ  wrote:
>
>> Hello Carl,
>>
>> Here is a message I sent to my team a few months ago. I hope this will be
>> helpful to you and more people around :). It might not be exhaustive and we
>> were moving from C*2.1 to C*3+ in this case, thus skipping C*2.2, but C*2.2
>> is similar to C*3.0 if I remember correctly in terms of metrics. Here it is
>> for what it's worth:
>>
>> Quite a few things changed between metric reporter in C* 2.1 and C*3.0.
>> - ColumnFamily --> Table
>> - XXpercentile --> pXX
>> - 1MinuteRate -->  m1_rate
>> - metric name before KS and Table names and some other changes of this
>> kind.
>> - ^ aggregations / aliases indexes changed because of this (using
>> graphite for example) ^
>> - ‘.value’ is not appended in the metric name anymore for gauges, nothing
>> instead.
>>
>> For example (graphite):
>>
>> From
>> aliasByNode(averageSeriesWithWildcards(cassandra.$env.$dc.$host.org.apache.cassandra.metrics.ColumnFamily.$ks.$table.ReadLatency.95percentile,
>> 2, 3), 1, 7, 8, 9)
>>
>> to
>> aliasByNode(averageSeriesWithWildcards(cassandra.$env.$dc.$host.org.apache.cassandra.metrics.Table.ReadLatency.$ks.$table.p95,
>> 2, 3), 1, 8, 9, 10)
>>
>> C*heers,
>> ---
>> Alain Rodriguez - @arodream - al...@thelastpickle.com
>> France / Spain
>>
>> The Last Pickle - Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> Le ven. 28 sept. 2018 à 20:38, Carl Mueller
>>  a écrit :
>>
>>> VERY NICE! Thank you very much
>>>
>>> On Fri, Sep 28, 2018 at 1:32 PM Lyuben Todorov <
>>> lyuben.todo...@instaclustr.com> wrote:
>>>
 Nothing as fancy as a matrix but a list of what JMX term can see.
 Link to the online diff here: https://www.diffchecker.com/G9FE9swS

 /lyubent

 On Fri, 28 Sep 2018 at 19:04, Carl Mueller
  wrote:

> It's my understanding that metrics got heavily re-namespaced in JMX
> for 2.2 from 2.1
>
> Did anyone ever make a migration matrix/guide for conversion of old
> metrics to new metrics?
>
>
>

Re: Re: Re: how to configure the Token Allocation Algorithm

[Varun Barala]

Hi,

Managing `initial_token` by yourself will give you more control over
scale-in and scale-out.
Let's say you have three node cluster with `num_token: 1`

And your initial range looks like:-

Datacenter: datacenter1
==
AddressRackStatus State   LoadOwns
 Token

 3074457345618258602
127.0.0.1  rack1   Up Normal  98.96 KiB   66.67%
 -9223372036854775808
127.0.0.2  rack1   Up Normal  98.96 KiB   66.67%
 -3074457345618258603
127.0.0.3  rack1   Up Normal  98.96 KiB   66.67%
 3074457345618258602

Now let's say you want to scale out the cluster to twice the current
throughput(means you are adding 3 more nodes)

If you are using AWS EBS volumes then you can use the same volumes and spin
three more nodes by selecting midpoints of existing ranges which means your
new nodes are already having data.
Once you have mounted volumes on your new nodes:-
* You need to delete every system table except schema related tables.
* You need to generate system/local table by yourself which has `Bootstrap
state` as completed and schema-version same as other existing nodes.
* You need to remove extra data on all the machines using cleanup commands

This is how you can scale out Cassandra cluster in the minutes. In case you
want to add nodes one by one then you need to write some small tool which
will always figure out the bigger range in the existing cluster and will
split it into the half.

However, I never tested it thoroughly but this should work conceptually. So
here we are taking advantage of the fact that we have volumes(data) for
the new node beforehand so we no need to bootstrap them.

Thanks & Regards,
Varun Barala

On Tue, Oct 2, 2018 at 2:31 PM onmstester onmstester 
wrote:

>
>
> Sent using Zoho Mail 
>
>
>  On Mon, 01 Oct 2018 18:36:03 +0330 *Alain RODRIGUEZ
> >* wrote 
>
> Hello again :),
>
> I thought a little bit more about this question, and I was actually
> wondering if something like this would work:
>
> Imagine 3 node cluster, and create them using:
> For the 3 nodes: `num_token: 4`
> Node 1: `intial_token: -9223372036854775808, -4611686018427387905, -2,
> 4611686018427387901`
> Node 2: `intial_token: -7686143364045646507, -3074457345618258604,
> 1537228672809129299, 6148914691236517202`
> Node 3: `intial_token: -6148914691236517206, -1537228672809129303,
> 3074457345618258600, 7686143364045646503`
>
>  If you know the initial size of your cluster, you can calculate the total
> number of tokens: number of nodes * vnodes and use the formula/python
> code above to get the tokens. Then use the first token for the first node,
> move to the second node, use the second token and repeat. In my case there
> is a total of 12 tokens (3 nodes, 4 tokens each)
> ```
> >>> number_of_tokens = 12
> >>> [str(((2**64 / number_of_tokens) * i) - 2**63) for i in
> range(number_of_tokens)]
> ['-9223372036854775808', '-7686143364045646507', '-6148914691236517206',
> '-4611686018427387905', '-3074457345618258604', '-1537228672809129303',
> '-2', '1537228672809129299', '3074457345618258600', '4611686018427387901',
> '6148914691236517202', '7686143364045646503']
> ```
>
>
> Using manual initial_token (your idea), how could i add a new node to a
> long running cluster (the procedure)?
>
>

Re: Re: Re: how to configure the Token Allocation Algorithm

[onmstester onmstester]

Sent using Zoho Mail  On Mon, 01 Oct 2018 18:36:03 +0330 Alain RODRIGUEZ 
 wrote  Hello again :), I thought a little bit more 
about this question, and I was actually wondering if something like this would 
work: Imagine 3 node cluster, and create them using: For the 3 nodes: 
`num_token: 4` Node 1: `intial_token: -9223372036854775808, 
-4611686018427387905, -2, 4611686018427387901` Node 2: `intial_token: 
-7686143364045646507, -3074457345618258604, 1537228672809129299, 
6148914691236517202` Node 3: `intial_token: -6148914691236517206, 
-1537228672809129303, 3074457345618258600, 7686143364045646503`  If you know 
the initial size of your cluster, you can calculate the total number of tokens: 
number of nodes * vnodes and use the formula/python code above to get the 
tokens. Then use the first token for the first node, move to the second node, 
use the second token and repeat. In my case there is a total of 12 tokens (3 
nodes, 4 tokens each) ``` >>> number_of_tokens = 12 >>> [str(((2**64 / 
number_of_tokens) * i) - 2**63) for i in range(number_of_tokens)] 
['-9223372036854775808', '-7686143364045646507', '-6148914691236517206', 
'-4611686018427387905', '-3074457345618258604', '-1537228672809129303', '-2', 
'1537228672809129299', '3074457345618258600', '4611686018427387901', 
'6148914691236517202', '7686143364045646503'] ``` Using manual initial_token 
(your idea), how could i add a new node to a long running cluster (the 
procedure)?