Hello Scott, Thanks a lot for the immediate answer. We use a semi automated procedure to do the upgrade of the SW in our systems which is done per datacenter. Our limitation is that if we want to rollback we need to rollback the Cassandra nodes from the whole datacenter. May I return to the alternatives: Would it be possible to add a new Datacenter of C*3 SW in a cluster with datacenters on c*3 and C*4 by limiting the data streaming only from the C*3 datacenters? Would it be possible to add a new Datacenter of C*4 SW in a cluster with datacenters on C*3 and C*4 by limiting the data streaming only from the C*4 datacenters? I am currently planning my tests so any initial statement would be really valuable for me to know what to expect and what to try out.
My purpose of investigating this is that I want to reduce the downtime of a datacenter in case of an upgrade failure. If I have the possibility to add a datacenter of either C*3 or C*4 SW in a cluster with datacenters of both C*3 or C*4, I will be able to handle its failure case independently and add the datacenter back to service quicker. My other alternatives in this case, is to upgrade the remaining datacenters from C*3 to C*4 and upgrade the “failed” datacenter to C*4 SW and add it back to the cluster. This has an increased time of operating with less datacenters and poses the risk when upgrade failures happen to multiple datacenters. BR MK From: C. Scott Andreas <sc...@paradoxica.net> Sent: October 26, 2023 10:43 To: user@cassandra.apache.org Cc: user@cassandra.apache.org; Michalis Kotsiouros (EXT) <michalis.kotsiouros....@ericsson.com> Subject: Re: Upgrade from C* 3 to C* 4 per datacenter The recommended approach to upgrading is to perform a replica-safe rolling restart of instances in each datacenter, one datacenter at a time. > In case of an upgrade failure, would it be possible to remove the data center > from the cluster, restore the datacenter to C*3 SW and add it back to cluster > which will contain datacenters in both C* 3 and C*4? Streaming and repair are not supported between 3.x and 4.x instances, so it will not be possible to bootstrap a datacenter as 3.x from nodes that are running 4.x. The approach above isn't an option and, in many topologies, may violate consistency or induce data loss. > Alternatively, could we remove the datacenter, perform the SW upgrade to C*4 > and then add it back to the cluster? You *could* do this if the datacenter is added back on 4.x, but it's not quite clear what it would accomplish. By far the safest and most tested upgrade path used by nearly everyone is a replica-safe rolling restart of instances in each datacenter, one datacenter at a time. Could you say more about the concerns you have with this upgrade path, or the worries you are hoping to mitigate? – Scott On Oct 26, 2023, at 8:32 AM, "Michalis Kotsiouros (EXT) via user" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> wrote: Hello Cassandra community, We are trying to upgrade our systems from Cassandra 3 to Cassandra 4. We plan to do this per data center. During the upgrade, a cluster with mixed SW levels is expected. At this point is it possible to perform topology changes? In case of an upgrade failure, would it be possible to remove the data center from the cluster, restore the datacenter to C*3 SW and add it back to cluster which will contain datacenters in both C* 3 and C*4? Alternatively, could we remove the datacenter, perform the SW upgrade to C*4 and then add it back to the cluster? Are there any suggestions or experiences regarding this fallback scenario? BR MK
