Re: best pointers to learn Cassandra maintenance

[Jeff Jirsa]

On Thu, Oct 8, 2020 at 5:31 AM Attila Wind  wrote:

> Hey Guys,
>
> We already started to feel that however Cassandra performance is awesome
> in the beginning over time
> - as more and more data is present in the tables,
> - more and more deletes creating tombstones,
> - cluster gets here and there not that well balanced
> performance can drop quickly and significantly...
>
> After ~1 year of learning curve we had to realize that time by time we run
> into things like "running repairs", "running compactions", understand
> tombstones (row and range), TTLs, etc etc becomes critical as data is
> growing.
> But on the other hand we also see often lots of warnings... Like "if you
> start Cassandra Reaper you can not stop doing that" ...
>
> I feel a bit confused now, and so far never ran into an article which
> really deeply explains: why?
> Why this? Why that? Why not this?
>
I know you're asking in general, but let me describe why it's hard - for
repair, in particular, there's a ton of nuance. In particular, there are
two types of repair (full and incremental), and then different scopes (-pr
for primary range, using start/end tokens for sub range, repairing all the
ranges on a host, etc).

With full repair, you compare all the data in a token range, stream
differences, and you're done. If you run the same command 30 seconds later,
it has to do the exact same amount of work.

With incremental repair, it uses clean/dirty bits on data files, and
optimizes so you dont have to scan as much data on subsequent runs. This
ALSO means you have 2 different sets of data files - clean and dirty - and
they won't ever compact together until you promote dirty files to clean
files! THAT is the magic bit of knowledge that most people don't describe
when they say "once you start running incremental repair, you can't stop".

If you're using reaper for full subrange repairs, you could stop at any
time. But if you're doing it for incremental, and you stop, you need to
unset all the repaired bits on the data files or you end up with data that
can't be compacted.

The time it takes to type out every single one of these surprising edge
cases / nuances is just too high for anyone to do it for free. Some books
try, many of the books are incomplete or out of date. It's a shame.

One day, hopefully, the database matures to a point where you don't need to
know how repair works in order to run a cluster. Oct 8 2020 is not that
day.


>
> So I think the time has come for us in the team to start focusing on these
> topics now. Invest time to better understanding. Really learn what "repair"
> means, and all consequences of it, etc
>
> So
> Does anyone have any "you must read it" recommendations around these "long
> term maintenance" topics?
>
Unfortunately, not really. There's some notes here
https://cassandra.apache.org/doc/latest/operating/index.html but it's
imperfect. May be good for people to keep adding docs.

> I mean really well explained blog post(s), article(s), book(s). Not some
> "half done" or  "I quickly write a post because it was too long ago when I
> blogged something..." things  :-)
>
> Good pointers would be appreciated!
>
> thanks
> --
> Attila Wind
>
> http://www.linkedin.com/in/attilaw
> Mobile: +49 176 43556932
>
>
>

Re: best pointers to learn Cassandra maintenance

[Erick Ramirez]

In Cassandra, "repair" refers to anti-entropy repairs. I think that's where
most of the confusion is. DBAs see the word "repair" and think it is a
one-off operation to "fix something broken". Users incorrectly think that
once it is fixed then there shouldn't be a need to repair again.

However in a distributed environment, the reality is that replicas can get
out of sync for whatever reason -- nodes going offline, nodes temporarily
unresponsive, nodes suffering from a hardware failure, etc. Entropy ensues.

It is necessary to keep the data consistent across the cluster so we run
anti-entropy repairs. The recommendation is that you run repairs at least
once every gc_grace_seconds (GCGS). GCGS by default is 10 days so a good
rule of thumb is to run repairs once a week.

Let me address some of the points you raised.

> ... we run into things like "running repairs", "running compactions",
> understand tombstones (row and range), TTLs, etc etc becomes critical as
> data is growing.
>
Compactions are part of the normal operation of Cassandra. You shouldn't
however be manually running compactions. If you are, something is wrong and
it's most likely a band-aid solution to an underlying problem you need to
address.

> But on the other hand we also see often lots of warnings... Like "if you
> start Cassandra Reaper you can not stop doing that" ...
>
As above, you need to run repairs regularly. It isn't a one-off operation.
Reaper is a good tool for managing repairs in an automated fashion.

Here are some useful resources on repairs in Cassandra:
- Repair document @ the Apache website -
https://cassandra.apache.org/doc/latest/operating/repair.html
- DataStax Academy video on Repair -
https://www.youtube.com/watch?v=5V5rGDTHs20
- YouTube playlist on DataStax Academy Cassandra Operations course -
https://www.youtube.com/playlist?list=PL2g2h-wyI4SrHMlHBJVe_or_Ryek2THgQ
- DataStax Doc on when to run repairs -
https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsRepairNodesWhen.html

Cheers!

>

Re: cassandra tracing's source_elapsed microseconds

[Erick Ramirez]

>
> Does the above indicate it took 20.696 milliseconds to run the last
> "Partition index with 0 entries found for sstable 6" activity?
>

Yes, that's correct. It's cumulative. And the last entry for "Request
complete" is the total time it took for the query. Cheers!

cassandra tracing's source_elapsed microseconds

[James A. Robinson]

Hi folks,

I've been looking at various articles on the TRACING ON output of cassandra.
I'm not finding a definitive description of what the output means.

https://docs.datastax.com/en/dse/6.7/cql/cql/cql_reference/cqlsh_commands/cqlshTracing.html

says "Note: The source_elapsed column value is the elapsed time of the
event on the source node in microseconds."

Am I correct in the understanding that the current row's source_elapsed
value minus the source_elapsed value of the previous row for the same
source node should tell me how long the current row took to execute?

As an example:

activity
  | timestamp  |
source
| source_elapsed | client
...
Bloom filter allows skipping sstable 396 [ReadStage-3]   |
2020-10-08 10:31:48.631001 | 10.220.50.148 |  402 |
10.220.50.148
Partition index with 0 entries found for sstable 382 [ReadStage-3] |
2020-10-08 10:31:48.636000 | 10.220.50.148 | 5819 |
10.220.50.148
Bloom filter allows skipping sstable 380 [ReadStage-3] |
2020-10-08 10:31:48.645000 | 10.220.50.148 |  14685 |
10.220.50.148
Bloom filter allows skipping sstable 14 [ReadStage-3]   |
2020-10-08 10:31:48.645000 | 10.220.50.148 |   14714 |
10.220.50.148
Partition index with 0 entries found for sstable 6 [ReadStage-3]|
2020-10-08 10:31:48.666000 | 10.220.50.148 |   35410 |
10.220.50.148
...

Does the above indicate it took 20.696 milliseconds to run the last
"Partition index with 0 entries found for sstable 6" activity?

Jim

Re: Tool for schema upgrades

[Alexander DEJANOVSKI]

I second Alex's recommendation.
We use https://github.com/patka/cassandra-migration to manage schema
migrations in Reaper and it has a consensus feature to prevent concurrent
migrations from clashing.

Cheers,

Alex

Le jeu. 8 oct. 2020 à 19:10, Alex Ott  a écrit :

> Hi
>
> Look at https://github.com/patka/cassandra-migration - it should be good.
>
> P.S. Here is the list of tools that I assembled over the years:
>
>- [ ] https://github.com/hhandoko/cassandra-migration
>- [ ] https://github.com/Contrast-Security-OSS/cassandra-migration
>- [ ] https://github.com/juxt/joplin
>- [ ] https://github.com/o19s/trireme
>- [ ] https://github.com/golang-migrate/migrate
>- [ ] https://github.com/Cobliteam/cassandra-migrate
>- [ ] https://github.com/patka/cassandra-migration
>- [ ] https://github.com/comeara/pillar
>
> On Thu, Oct 8, 2020 at 5:45 PM Paul Chandler  wrote:
>
>> Hi all,
>>
>> Can anyone recommend a tool to perform schema DDL upgrades, that follows
>> best practice to ensure you don’t get schema mismatches if running multiple
>> upgrade statements in one migration ?
>>
>> Thanks
>>
>> Paul
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>
> --
> With best wishes,Alex Ott
> http://alexott.net/
> Twitter: alexott_en (English), alexott (Russian)
>

Re: Tool for schema upgrades

[Alex Ott]

Hi

Look at https://github.com/patka/cassandra-migration - it should be good.

P.S. Here is the list of tools that I assembled over the years:

   - [ ] https://github.com/hhandoko/cassandra-migration
   - [ ] https://github.com/Contrast-Security-OSS/cassandra-migration
   - [ ] https://github.com/juxt/joplin
   - [ ] https://github.com/o19s/trireme
   - [ ] https://github.com/golang-migrate/migrate
   - [ ] https://github.com/Cobliteam/cassandra-migrate
   - [ ] https://github.com/patka/cassandra-migration
   - [ ] https://github.com/comeara/pillar

On Thu, Oct 8, 2020 at 5:45 PM Paul Chandler  wrote:

> Hi all,
>
> Can anyone recommend a tool to perform schema DDL upgrades, that follows
> best practice to ensure you don’t get schema mismatches if running multiple
> upgrade statements in one migration ?
>
> Thanks
>
> Paul
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

-- 
With best wishes,Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)

Tool for schema upgrades

[Paul Chandler]

Hi all,

Can anyone recommend a tool to perform schema DDL upgrades, that follows best 
practice to ensure you don’t get schema mismatches if running multiple upgrade 
statements in one migration ?

Thanks 

Paul
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Reducing no. of nodes to 0 without losing data

[Manu Chadha]

Tried persistence as well (just table schemas, no data). It works as well 
across restarts. Next challenge for me is to find out how to backup the data in 
case there is a cluster wide data loss.

Sent from Mail for Windows 10

From: Manu Chadha
Sent: 08 October 2020 13:33
To: user@cassandra.apache.org
Subject: RE: Reducing no. of nodes to 0 without losing data

Dear Christopher

I tried the answer and it works (at least the stopping bit). I still need to 
test persistence bit which to be honest might take time considering that I am 
more rubbish in K8S than Cassandra.  I had created an question on SO. Please 
feel free to provide your answer there.

https://stackoverflow.com/questions/64253051/temporarily-stopping-k8s-and-cassandra-cluster?noredirect=1#comment113633984_64253051

thanks
Manu

Sent from Mail for Windows 10

From: Christopher Bradford
Sent: 08 October 2020 05:47
To: user@cassandra.apache.org
Subject: Re: Reducing no. of nodes to 0 without losing data

cass-operator has a parameter spec.stopped (
https://github.com/datastax/cass-operator/blob/master/operator/example-cassdc-yaml/cassandra-3.11.x/example-cassdc-full.yaml#L65)
 when set to true the underlying stateful sets are scaled down to 0 while 
keeping the persistent volumes. This allows you to reduce your k8s cluster size 
while not losing data. When you want to bring the cluster back up change the 
parameter back to false.

On Thu, Oct 8, 2020 at 12:40 AM Oleksandr Shulgin 
mailto:oleksandr.shul...@zalando.de>> wrote:
On Thu, Oct 8, 2020 at 12:21 AM Manu Chadha 
mailto:manu.cha...@hotmail.com>> wrote:
Hi

I have created a Cassandra cluster on Kubernetes using cass-operator on gcp. It 
is for my personal experimentation. To avoid incurring cost, I want to stop the 
cluster when I am not using it and start it when I need it without losing data. 
Is there a way to do so? Would setting number of size to 0 in 
example-cassdc-minimal.yaml stop the compute resources without losing data? If 
I change the size to 3 again later, would the existing data be picked?

Depending if the operator is going to accept the size as 0 at all, but most 
probably not with the following policy in your storage class, as in the 
example[1]:

reclaimPolicy: Delete

You need some persistent storage and a suitable reclaim policy.

[1]: 
https://docs.datastax.com/en/cass-operator/doc/cass-operator/cassOperatorCloserLookConfiguration.html#CreateandapplyaStorageClass

Regards,
--
Alex
--

Christopher Bradford

RE: Reducing no. of nodes to 0 without losing data

[Manu Chadha]

Dear Christopher

I tried the answer and it works (at least the stopping bit). I still need to 
test persistence bit which to be honest might take time considering that I am 
more rubbish in K8S than Cassandra.  I had created an question on SO. Please 
feel free to provide your answer there.

https://stackoverflow.com/questions/64253051/temporarily-stopping-k8s-and-cassandra-cluster?noredirect=1#comment113633984_64253051

thanks
Manu

Sent from Mail for Windows 10

From: Christopher Bradford
Sent: 08 October 2020 05:47
To: user@cassandra.apache.org
Subject: Re: Reducing no. of nodes to 0 without losing data

cass-operator has a parameter spec.stopped (
https://github.com/datastax/cass-operator/blob/master/operator/example-cassdc-yaml/cassandra-3.11.x/example-cassdc-full.yaml#L65)
 when set to true the underlying stateful sets are scaled down to 0 while 
keeping the persistent volumes. This allows you to reduce your k8s cluster size 
while not losing data. When you want to bring the cluster back up change the 
parameter back to false.

On Thu, Oct 8, 2020 at 12:40 AM Oleksandr Shulgin 
mailto:oleksandr.shul...@zalando.de>> wrote:
On Thu, Oct 8, 2020 at 12:21 AM Manu Chadha 
mailto:manu.cha...@hotmail.com>> wrote:
Hi

I have created a Cassandra cluster on Kubernetes using cass-operator on gcp. It 
is for my personal experimentation. To avoid incurring cost, I want to stop the 
cluster when I am not using it and start it when I need it without losing data. 
Is there a way to do so? Would setting number of size to 0 in 
example-cassdc-minimal.yaml stop the compute resources without losing data? If 
I change the size to 3 again later, would the existing data be picked?

Depending if the operator is going to accept the size as 0 at all, but most 
probably not with the following policy in your storage class, as in the 
example[1]:

reclaimPolicy: Delete

You need some persistent storage and a suitable reclaim policy.

[1]: 
https://docs.datastax.com/en/cass-operator/doc/cass-operator/cassOperatorCloserLookConfiguration.html#CreateandapplyaStorageClass

Regards,
--
Alex
--

Christopher Bradford

best pointers to learn Cassandra maintenance

[Attila Wind]

Hey Guys,

We already started to feel that however Cassandra performance is awesome 
in the beginning over time

- as more and more data is present in the tables,
- more and more deletes creating tombstones,
- cluster gets here and there not that well balanced
performance can drop quickly and significantly...

After ~1 year of learning curve we had to realize that time by time we 
run into things like "running repairs", "running compactions", 
understand tombstones (row and range), TTLs, etc etc becomes critical as 
data is growing.
But on the other hand we also see often lots of warnings... Like "if you 
start Cassandra Reaper you can not stop doing that" ...


I feel a bit confused now, and so far never ran into an article which 
really deeply explains: why?

Why this? Why that? Why not this?

So I think the time has come for us in the team to start focusing on 
these topics now. Invest time to better understanding. Really learn what 
"repair" means, and all consequences of it, etc


So
Does anyone have any "you must read it" recommendations around these 
"long term maintenance" topics?
I mean really well explained blog post(s), article(s), book(s). Not some 
"half done" or  "I quickly write a post because it was too long ago when 
I blogged something..." things  :-)


Good pointers would be appreciated!

thanks

--
Attila Wind

http://www.linkedin.com/in/attilaw
Mobile: +49 176 43556932

RE: Reducing no. of nodes to 0 without losing data

[Manu Chadha]

The operator doesn’t accept 0. Changing the size in 
example-cass-dc-minimal.yaml doesn't work as I get error `The 
CassandraDatacenter "dc1" is invalid: spec.size: Invalid value: 1: spec.size in 
body should be greater than or equal to 1`

Sent from Mail for Windows 10

From: Oleksandr Shulgin
Sent: 08 October 2020 05:40
To: User
Subject: Re: Reducing no. of nodes to 0 without losing data

On Thu, Oct 8, 2020 at 12:21 AM Manu Chadha 
mailto:manu.cha...@hotmail.com>> wrote:
Hi

I have created a Cassandra cluster on Kubernetes using cass-operator on gcp. It 
is for my personal experimentation. To avoid incurring cost, I want to stop the 
cluster when I am not using it and start it when I need it without losing data. 
Is there a way to do so? Would setting number of size to 0 in 
example-cassdc-minimal.yaml stop the compute resources without losing data? If 
I change the size to 3 again later, would the existing data be picked?

Depending if the operator is going to accept the size as 0 at all, but most 
probably not with the following policy in your storage class, as in the 
example[1]:

reclaimPolicy: Delete

You need some persistent storage and a suitable reclaim policy.

[1]: 
https://docs.datastax.com/en/cass-operator/doc/cass-operator/cassOperatorCloserLookConfiguration.html#CreateandapplyaStorageClass

Regards,
--
Alex