RE: Datacenter decommissioning on Cassandra 4.1.4

[Michalis Kotsiouros (EXT) via user]

Hello Jon and Jeff,
Thanks a lot for your replies.
I completely get your points.
Some more clarification about my issue.
When trying to update the Replication before the decommission, I get the 
following error message when I remove the replication for system_auth kesypace.
ConfigurationException: Following datacenters have active nodes and must be 
present in replication options for keyspace system_auth: [datacenter1]

This error message does not appear in the rest of the application keyspaces.
So, may I change the procedure to:

  1.  Make sure no clients are still writing to any nodes in the datacenter.
  2.  Run a full repair with nodetool repair.
  3.  Change all keyspaces so they no longer reference the datacenter being 
removed apart from system_auth keyspace.
  4.  Run nodetool decommission using the --force option on every node in the 
datacenter being removed.
  5.  Change system_auth keyspace so they no longer reference the datacenter 
being removed.
BR
MK



From: Jeff Jirsa 
Sent: April 08, 2024 17:19
To: cassandra 
Cc: Michalis Kotsiouros (EXT) 
Subject: Re: Datacenter decommissioning on Cassandra 4.1.4

To Jon’s point, if you remove from replication after step 1 or step 2 (probably 
step 2 if your goal is to be strictly correct), the nodetool decommission phase 
becomes almost a no-op.

If you use the order below, the last nodes to decommission will cause those 
surviving machines to run out of space (assuming you have more than a few nodes 
to start)




On Apr 8, 2024, at 6:58 AM, Jon Haddad 
mailto:j...@jonhaddad.com>> wrote:

You shouldn’t decom an entire DC before removing it from replication.

—

Jon Haddad
Rustyrazorblade Consulting
rustyrazorblade.com


On Mon, Apr 8, 2024 at 6:26 AM Michalis Kotsiouros (EXT) via user 
mailto:user@cassandra.apache.org>> wrote:
Hello community,
In our deployments, we usually rebuild the Cassandra datacenters for 
maintenance or recovery operations.
The procedure used since the days of Cassandra 3.x was the one documented in 
datastax documentation. Decommissioning a datacenter | Apache Cassandra 3.x 
(datastax.com)
After upgrading to Cassandra 4.1.4, we have realized that there are some 
stricter rules that do not allo to remove the replication when active Cassandra 
nodes still exist in a datacenter.
This check makes the above-mentioned procedure obsolete.
I am thinking to use the following as an alternative:

  1.  Make sure no clients are still writing to any nodes in the datacenter.
  2.  Run a full repair with nodetool repair.
  3.  Run nodetool decommission using the --force option on every node in the 
datacenter being removed.
  4.  Change all keyspaces so they no longer reference the datacenter being 
removed.

What is the procedure followed by other users? Do you see any risk following 
the proposed procedure?

BR
MK

Re: Datacenter decommissioning on Cassandra 4.1.4

[Jeff Jirsa]

To Jon’s point, if you remove from replication after step 1 or step 2 (probably 
step 2 if your goal is to be strictly correct), the nodetool decommission phase 
becomes almost a no-op. 

If you use the order below, the last nodes to decommission will cause those 
surviving machines to run out of space (assuming you have more than a few nodes 
to start)



> On Apr 8, 2024, at 6:58 AM, Jon Haddad  wrote:
> 
> You shouldn’t decom an entire DC before removing it from replication.
> 
> —
> 
> Jon Haddad
> Rustyrazorblade Consulting
> rustyrazorblade.com 
> 
> 
> On Mon, Apr 8, 2024 at 6:26 AM Michalis Kotsiouros (EXT) via user 
> mailto:user@cassandra.apache.org>> wrote:
>> Hello community,
>> 
>> In our deployments, we usually rebuild the Cassandra datacenters for 
>> maintenance or recovery operations.
>> 
>> The procedure used since the days of Cassandra 3.x was the one documented in 
>> datastax documentation. Decommissioning a datacenter | Apache Cassandra 3.x 
>> (datastax.com) 
>> 
>> After upgrading to Cassandra 4.1.4, we have realized that there are some 
>> stricter rules that do not allo to remove the replication when active 
>> Cassandra nodes still exist in a datacenter.
>> 
>> This check makes the above-mentioned procedure obsolete.
>> 
>> I am thinking to use the following as an alternative:
>> 
>> Make sure no clients are still writing to any nodes in the datacenter.
>> Run a full repair with nodetool repair.
>> Run nodetool decommission using the --force option on every node in the 
>> datacenter being removed.
>> Change all keyspaces so they no longer reference the datacenter being 
>> removed.
>>  
>> 
>> What is the procedure followed by other users? Do you see any risk following 
>> the proposed procedure?
>> 
>>  
>> 
>> BR
>> 
>> MK
>> 

Re: Datacenter decommissioning on Cassandra 4.1.4

[Jon Haddad]

You shouldn’t decom an entire DC before removing it from replication.

—

Jon Haddad
Rustyrazorblade Consulting
rustyrazorblade.com


On Mon, Apr 8, 2024 at 6:26 AM Michalis Kotsiouros (EXT) via user <
user@cassandra.apache.org> wrote:

> Hello community,
>
> In our deployments, we usually rebuild the Cassandra datacenters for
> maintenance or recovery operations.
>
> The procedure used since the days of Cassandra 3.x was the one documented
> in datastax documentation. Decommissioning a datacenter | Apache
> Cassandra 3.x (datastax.com)
> 
>
> After upgrading to Cassandra 4.1.4, we have realized that there are some
> stricter rules that do not allo to remove the replication when active
> Cassandra nodes still exist in a datacenter.
>
> This check makes the above-mentioned procedure obsolete.
>
> I am thinking to use the following as an alternative:
>
>1. Make sure no clients are still writing to any nodes in the
>datacenter.
>2. Run a full repair with nodetool repair.
>3. Run nodetool decommission using the --force option on every node in
>the datacenter being removed.
>4. Change all keyspaces so they no longer reference the datacenter
>being removed.
>
>
>
> What is the procedure followed by other users? Do you see any risk
> following the proposed procedure?
>
>
>
> BR
>
> MK
>

Datacenter decommissioning on Cassandra 4.1.4

[Michalis Kotsiouros (EXT) via user]

Hello community,
In our deployments, we usually rebuild the Cassandra datacenters for 
maintenance or recovery operations.
The procedure used since the days of Cassandra 3.x was the one documented in 
datastax documentation. Decommissioning a datacenter | Apache Cassandra 3.x 
(datastax.com)
After upgrading to Cassandra 4.1.4, we have realized that there are some 
stricter rules that do not allo to remove the replication when active Cassandra 
nodes still exist in a datacenter.
This check makes the above-mentioned procedure obsolete.
I am thinking to use the following as an alternative:

  1.  Make sure no clients are still writing to any nodes in the datacenter.
  2.  Run a full repair with nodetool repair.
  3.  Run nodetool decommission using the --force option on every node in the 
datacenter being removed.
  4.  Change all keyspaces so they no longer reference the datacenter being 
removed.

What is the procedure followed by other users? Do you see any risk following 
the proposed procedure?

BR
MK