Re: Cassandra 4.0 and changing DC setting

[Kane Wilson]

Great thanks for testing this Paul. You're likely the first one to actually
try it on 4.0. Albeit I still think we should have a flag to ALTER to do
this for you... not ideal to have users messing with system tables.

raft.so - Cassandra consulting, support, and managed services


On Mon, Feb 22, 2021 at 10:58 PM Paul Chandler  wrote:

> Yes, I am only running this on test clusters, I don’t run anything like
> this without lots of tests first.
>
> Anyway this worked well, so thanks for the info.
>
> For anyone else who needs this, the cql statement to do this was:
>
> insert into system_schema.keyspaces ( keyspace_name , durable_writes,
> replication ) values ( 'system_auth', true, {'OLD_DC': '3', ’NEW_DC': '3',
> 'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy'} );
>
> This will then allow you to log in after the nodes come back up, and the
> other keyspaces can be changed as normal afterwards.
>
> Thanks for you help.
>
> Paul
>
>
> On 21 Feb 2021, at 22:30, Kane Wilson  wrote:
>
> Make sure you test it on a practice cluster. Messing with the system
> tables is risky business!
>
> raft.so - Cassandra consulting, support, and managed services
>
>
> On Sun, Feb 21, 2021 at 11:12 PM Paul Chandler  wrote:
>
>> Hi Kane,
>>
>> That sounds a good idea, I will give it a try on Monday.
>>
>> Thanks
>>
>> Paul
>>
>> On 21 Feb 2021, at 11:33, Kane Wilson  wrote:
>>
>> There has been proposals to add a force/unsafe flag to alter DC but it
>> hasn't been actioned and at this rate seems unlikely to make it into 4.0.
>> There is however a workaround, albeit not very user friendly. You should be
>> able to modify the system_schema tables directly to do your DC updates. I
>> am yet to test it out myself so don't know the updates you'd need to make
>> but you should be able to get a good idea by querying, doing updates and
>> observing the effect.
>>
>> raft.so - Cassandra consulting, support, managed services
>>
>> On Sat., 20 Feb. 2021, 02:29 Paul Chandler,  wrote:
>>
>>> All,
>>>
>>> We have a use case where we need to change the datacenter name for a
>>> cassandra cluster, we have a script to do this that involves a short
>>> downtime. This does the following
>>>
>>>
>>> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’
>>> : ‘3’, ’NEW_DC”: ‘3’  }
>>> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each
>>> node
>>> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
>>> 4) Stop all nodes
>>> 5) Start each seed node, then start the rest of the nodes
>>> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”:
>>> ‘3’  }
>>>
>>> In 3.11.x this all works fine and the cluster is good to be used again
>>> after step 6.
>>>
>>> However in 4.0 step 1 is now blocked by the following change "Cassandra
>>> will no longer allow invalid keyspace replication options, such as invalid
>>> datacenter names for NetworkTopologyStrategy”
>>>
>>> If you skip step 1) then when the nodes come back up, you cannot login
>>> because the system_auth keyspace still has a replication factor of ‘OLD_DC’
>>> : ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be
>>> accessed.
>>>
>>> Is there a way around this to change the name of the datacenter?
>>>
>>>
>>> Thanks
>>>
>>> Paul
>>>
>>>
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>
>>
>

Re: Cassandra 4.0 and changing DC setting

[Paul Chandler]

Yes, I am only running this on test clusters, I don’t run anything like this 
without lots of tests first.

Anyway this worked well, so thanks for the info.

For anyone else who needs this, the cql statement to do this was:

insert into system_schema.keyspaces ( keyspace_name , durable_writes, 
replication ) values ( 'system_auth', true, {'OLD_DC': '3', ’NEW_DC': '3', 
'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy'} );

This will then allow you to log in after the nodes come back up, and the other 
keyspaces can be changed as normal afterwards.

Thanks for you help.

Paul


> On 21 Feb 2021, at 22:30, Kane Wilson  wrote:
> 
> Make sure you test it on a practice cluster. Messing with the system tables 
> is risky business!
> 
> raft.so  - Cassandra consulting, support, and managed 
> services
> 
> 
> On Sun, Feb 21, 2021 at 11:12 PM Paul Chandler  > wrote:
> Hi Kane,
> 
> That sounds a good idea, I will give it a try on Monday.
> 
> Thanks 
> 
> Paul
> 
>> On 21 Feb 2021, at 11:33, Kane Wilson mailto:k...@raft.so>> 
>> wrote:
>> 
>> There has been proposals to add a force/unsafe flag to alter DC but it 
>> hasn't been actioned and at this rate seems unlikely to make it into 4.0. 
>> There is however a workaround, albeit not very user friendly. You should be 
>> able to modify the system_schema tables directly to do your DC updates. I am 
>> yet to test it out myself so don't know the updates you'd need to make but 
>> you should be able to get a good idea by querying, doing updates and 
>> observing the effect.
>> 
>> raft.so - Cassandra consulting, support, managed services
>> 
>> On Sat., 20 Feb. 2021, 02:29 Paul Chandler, > > wrote:
>> All,
>> 
>> We have a use case where we need to change the datacenter name for a 
>> cassandra cluster, we have a script to do this that involves a short 
>> downtime. This does the following 
>> 
>> 
>> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’ : 
>> ‘3’, ’NEW_DC”: ‘3’  }
>> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each 
>> node 
>> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
>> 4) Stop all nodes
>> 5) Start each seed node, then start the rest of the nodes
>> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”: ‘3’  }
>> 
>> In 3.11.x this all works fine and the cluster is good to be used again after 
>> step 6.
>> 
>> However in 4.0 step 1 is now blocked by the following change "Cassandra will 
>> no longer allow invalid keyspace replication options, such as invalid 
>> datacenter names for NetworkTopologyStrategy”
>> 
>> If you skip step 1) then when the nodes come back up, you cannot login 
>> because the system_auth keyspace still has a replication factor of ‘OLD_DC’ 
>> : ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be 
>> accessed.
>> 
>> Is there a way around this to change the name of the datacenter?
>> 
>> 
>> Thanks 
>> 
>> Paul
>> 
>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>> 
>> For additional commands, e-mail: user-h...@cassandra.apache.org 
>> 
>> 
> 

Re: Cassandra 4.0 and changing DC setting

[Kane Wilson]

Make sure you test it on a practice cluster. Messing with the system tables
is risky business!

raft.so - Cassandra consulting, support, and managed services


On Sun, Feb 21, 2021 at 11:12 PM Paul Chandler  wrote:

> Hi Kane,
>
> That sounds a good idea, I will give it a try on Monday.
>
> Thanks
>
> Paul
>
> On 21 Feb 2021, at 11:33, Kane Wilson  wrote:
>
> There has been proposals to add a force/unsafe flag to alter DC but it
> hasn't been actioned and at this rate seems unlikely to make it into 4.0.
> There is however a workaround, albeit not very user friendly. You should be
> able to modify the system_schema tables directly to do your DC updates. I
> am yet to test it out myself so don't know the updates you'd need to make
> but you should be able to get a good idea by querying, doing updates and
> observing the effect.
>
> raft.so - Cassandra consulting, support, managed services
>
> On Sat., 20 Feb. 2021, 02:29 Paul Chandler,  wrote:
>
>> All,
>>
>> We have a use case where we need to change the datacenter name for a
>> cassandra cluster, we have a script to do this that involves a short
>> downtime. This does the following
>>
>>
>> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’ :
>> ‘3’, ’NEW_DC”: ‘3’  }
>> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each
>> node
>> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
>> 4) Stop all nodes
>> 5) Start each seed node, then start the rest of the nodes
>> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”:
>> ‘3’  }
>>
>> In 3.11.x this all works fine and the cluster is good to be used again
>> after step 6.
>>
>> However in 4.0 step 1 is now blocked by the following change "Cassandra
>> will no longer allow invalid keyspace replication options, such as invalid
>> datacenter names for NetworkTopologyStrategy”
>>
>> If you skip step 1) then when the nodes come back up, you cannot login
>> because the system_auth keyspace still has a replication factor of ‘OLD_DC’
>> : ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be
>> accessed.
>>
>> Is there a way around this to change the name of the datacenter?
>>
>>
>> Thanks
>>
>> Paul
>>
>>
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

Re: Cassandra 4.0 and changing DC setting

[Paul Chandler]

Hi Kane,

That sounds a good idea, I will give it a try on Monday.

Thanks 

Paul

> On 21 Feb 2021, at 11:33, Kane Wilson  wrote:
> 
> There has been proposals to add a force/unsafe flag to alter DC but it hasn't 
> been actioned and at this rate seems unlikely to make it into 4.0. There is 
> however a workaround, albeit not very user friendly. You should be able to 
> modify the system_schema tables directly to do your DC updates. I am yet to 
> test it out myself so don't know the updates you'd need to make but you 
> should be able to get a good idea by querying, doing updates and observing 
> the effect.
> 
> raft.so - Cassandra consulting, support, managed services
> 
> On Sat., 20 Feb. 2021, 02:29 Paul Chandler,  > wrote:
> All,
> 
> We have a use case where we need to change the datacenter name for a 
> cassandra cluster, we have a script to do this that involves a short 
> downtime. This does the following 
> 
> 
> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’ : 
> ‘3’, ’NEW_DC”: ‘3’  }
> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each node 
> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
> 4) Stop all nodes
> 5) Start each seed node, then start the rest of the nodes
> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”: ‘3’  }
> 
> In 3.11.x this all works fine and the cluster is good to be used again after 
> step 6.
> 
> However in 4.0 step 1 is now blocked by the following change "Cassandra will 
> no longer allow invalid keyspace replication options, such as invalid 
> datacenter names for NetworkTopologyStrategy”
> 
> If you skip step 1) then when the nodes come back up, you cannot login 
> because the system_auth keyspace still has a replication factor of ‘OLD_DC’ : 
> ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be 
> accessed.
> 
> Is there a way around this to change the name of the datacenter?
> 
> 
> Thanks 
> 
> Paul
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 
> 

Re: Cassandra 4.0 and changing DC setting

[Kane Wilson]

There has been proposals to add a force/unsafe flag to alter DC but it
hasn't been actioned and at this rate seems unlikely to make it into 4.0.
There is however a workaround, albeit not very user friendly. You should be
able to modify the system_schema tables directly to do your DC updates. I
am yet to test it out myself so don't know the updates you'd need to make
but you should be able to get a good idea by querying, doing updates and
observing the effect.

raft.so - Cassandra consulting, support, managed services

On Sat., 20 Feb. 2021, 02:29 Paul Chandler,  wrote:

> All,
>
> We have a use case where we need to change the datacenter name for a
> cassandra cluster, we have a script to do this that involves a short
> downtime. This does the following
>
>
> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’ :
> ‘3’, ’NEW_DC”: ‘3’  }
> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each
> node
> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
> 4) Stop all nodes
> 5) Start each seed node, then start the rest of the nodes
> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”:
> ‘3’  }
>
> In 3.11.x this all works fine and the cluster is good to be used again
> after step 6.
>
> However in 4.0 step 1 is now blocked by the following change "Cassandra
> will no longer allow invalid keyspace replication options, such as invalid
> datacenter names for NetworkTopologyStrategy”
>
> If you skip step 1) then when the nodes come back up, you cannot login
> because the system_auth keyspace still has a replication factor of ‘OLD_DC’
> : ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be
> accessed.
>
> Is there a way around this to change the name of the datacenter?
>
>
> Thanks
>
> Paul
>
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>