Re: 4.0 upgrade

[Bowen Song via user]

You should not make DDL (e.g. TRUNCATE, ALTER TABLE) or DCL (e.g. GRANT, 
ALTER ROLE) operations or run repair on a mixed version cluster. Source: 
https://www.datastax.com/learn/whats-new-for-cassandra-4/migrating-cassandra-4x


You should also ensure the gc_grace_seconds value is large enough to 
allow for the time to upgrade DC1, wait, upgrade DC2 and then complete a 
repair, or you may end up with resurrected data.


You also must ensure you do not enable any new features on new version 
nodes in a mixed version cluster. You may enable new features after all 
nodes in the cluster are upgraded.


On 07/07/2023 20:50, Runtian Liu wrote:

Hi,

We are upgrading our Cassandra clusters from 3.0.27 to 4.0.6 and we 
observed some error related to repair: j.l.IllegalArgumentException: 
Unknown verb id 32


We have two datacenters for each Cassandra cluster and when we are 
doing an upgrade, we want to upgrade 1 datacenter first and monitor 
the upgrade datacenter for some time (1 week) to make sure there is no 
issue, then we will upgrade the second datacenter for that cluster.


We have some automated repair jobs running, is it expected to have 
repair stuck if we have 1 datacenter on 4.0 and 1 datacenter on 3.0?


Do you have any suggestions on how we should do the upgrade, is 
waiting for 1 week between two datacenters too long?


Thanks,
Runtian

Re: 4.0 upgrade

[manish khandelwal]

Yes repairs are prohibited in mixed version cluster. If you want to monitor
please disable repairs till complete upgrade is finished

On Sat, Jul 8, 2023, 01:21 Runtian Liu  wrote:

> Hi,
>
> We are upgrading our Cassandra clusters from 3.0.27 to 4.0.6 and we
> observed some error related to repair:  j.l.IllegalArgumentException:
> Unknown verb id 32
>
> We have two datacenters for each Cassandra cluster and when we are doing
> an upgrade, we want to upgrade 1 datacenter first and monitor the upgrade
> datacenter for some time (1 week) to make sure there is no issue, then we
> will upgrade the second datacenter for that cluster.
>
> We have some automated repair jobs running, is it expected to have repair
> stuck if we have 1 datacenter on 4.0 and 1 datacenter on 3.0?
>
> Do you have any suggestions on how we should do the upgrade, is waiting
> for 1 week between two datacenters too long?
>
> Thanks,
> Runtian
>
>

4.0 upgrade

[Runtian Liu]

Hi,

We are upgrading our Cassandra clusters from 3.0.27 to 4.0.6 and we
observed some error related to repair:  j.l.IllegalArgumentException:
Unknown verb id 32

We have two datacenters for each Cassandra cluster and when we are doing an
upgrade, we want to upgrade 1 datacenter first and monitor the upgrade
datacenter for some time (1 week) to make sure there is no issue, then we
will upgrade the second datacenter for that cluster.

We have some automated repair jobs running, is it expected to have repair
stuck if we have 1 datacenter on 4.0 and 1 datacenter on 3.0?

Do you have any suggestions on how we should do the upgrade, is waiting for
1 week between two datacenters too long?

Thanks,
Runtian

Re: Cassandra 4.0 upgrade - Upgradesstables

[Jim Shaw]

Though it is not required to run upgradesstables, but upgradesstables -a
will re-write the file to kick out tombstones, in sizeTieredcompaction, the
largest files may stay a long time to wait for the next compaction to
kick out tombstones.
So it really depends,  to run it or not,  usually upgrades have a change
window, applications may be no load or less load, why don't take the chance
to run it.

Regards,

Jim

On Tue, Aug 16, 2022 at 3:17 PM Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Thank you
>
> On Tue, Aug 16, 2022 at 11:48 AM C. Scott Andreas 
> wrote:
>
>> No downside at all for 3.x -> 4.x (however, Cassandra 3.x reading 2.1
>> SSTables incurred a performance hit).
>>
>> Many users of Cassandra don't run upgradesstables after 3.x -> 4.x
>> upgrades at all. It's not necessary to run until a hypothetical future time
>> if/when support for reading Cassandra 3.x SSTables is removed from
>> Cassandra. One of the most common reasons to avoid running upgradesstables
>> is because doing so causes 100% churn of the data files, meaning your
>> backup processes will need to upload a full copy of the data. Allowing
>> SSTables to organically churn into the new version via compaction avoids
>> this.
>>
>> If you're upgrading from 3.x to 4.x, don't feel like you have to - but it
>> does avoid the need to run upgradesstables in a hypothetical distant future.
>>
>> – Scott
>>
>> On Aug 16, 2022, at 6:32 AM, Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>>
>> Thank you Erick,
>>
>> > it is going to be single-threaded by default so it will take a while to
>> get through all the sstables on dense nodes
>> Is there any downside if the upgradesstables take longer (example 1-2
>> days), other than I/O?
>>
>> Also when is the upgradesstable get triggered? after every node is
>> upgraded or it will kick in only when all the nodes in the cluster upgraded
>> to 4.0.x?
>>
>> On Tue, Aug 16, 2022 at 2:12 AM Erick Ramirez 
>> wrote:
>>
>>> As convenient as it is, there are a few caveats and it isn't a silver
>>> bullet. The automatic feature will only kick in if there are no other
>>> compactions scheduled. Also, it is going to be single-threaded by default
>>> so it will take a while to get through all the sstables on dense nodes.
>>>
>>> In contrast, you'll have a bit more control if you manually upgrade the
>>> sstables. For example, you can schedule the upgrade during low traffic
>>> periods so reads are not competing with compactions for IO. Cheers!
>>>


>>

Re: Cassandra 4.0 upgrade - Upgradesstables

[Jai Bheemsen Rao Dhanwada]

Thank you

On Tue, Aug 16, 2022 at 11:48 AM C. Scott Andreas 
wrote:

> No downside at all for 3.x -> 4.x (however, Cassandra 3.x reading 2.1
> SSTables incurred a performance hit).
>
> Many users of Cassandra don't run upgradesstables after 3.x -> 4.x
> upgrades at all. It's not necessary to run until a hypothetical future time
> if/when support for reading Cassandra 3.x SSTables is removed from
> Cassandra. One of the most common reasons to avoid running upgradesstables
> is because doing so causes 100% churn of the data files, meaning your
> backup processes will need to upload a full copy of the data. Allowing
> SSTables to organically churn into the new version via compaction avoids
> this.
>
> If you're upgrading from 3.x to 4.x, don't feel like you have to - but it
> does avoid the need to run upgradesstables in a hypothetical distant future.
>
> – Scott
>
> On Aug 16, 2022, at 6:32 AM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>
> Thank you Erick,
>
> > it is going to be single-threaded by default so it will take a while to
> get through all the sstables on dense nodes
> Is there any downside if the upgradesstables take longer (example 1-2
> days), other than I/O?
>
> Also when is the upgradesstable get triggered? after every node is
> upgraded or it will kick in only when all the nodes in the cluster upgraded
> to 4.0.x?
>
> On Tue, Aug 16, 2022 at 2:12 AM Erick Ramirez 
> wrote:
>
>> As convenient as it is, there are a few caveats and it isn't a silver
>> bullet. The automatic feature will only kick in if there are no other
>> compactions scheduled. Also, it is going to be single-threaded by default
>> so it will take a while to get through all the sstables on dense nodes.
>>
>> In contrast, you'll have a bit more control if you manually upgrade the
>> sstables. For example, you can schedule the upgrade during low traffic
>> periods so reads are not competing with compactions for IO. Cheers!
>>
>>>
>>>
>

Re: Cassandra 4.0 upgrade - Upgradesstables

[C. Scott Andreas]

No downside at all for 3.x -> 4.x (however, Cassandra 3.x reading 2.1 SSTables incurred a 
performance hit).Many users of Cassandra don't run upgradesstables after 3.x -> 4.x upgrades at 
all. It's not necessary to run until a hypothetical future time if/when support for reading 
Cassandra 3.x SSTables is removed from Cassandra. One of the most common reasons to avoid running 
upgradesstables is because doing so causes 100% churn of the data files, meaning your backup 
processes will need to upload a full copy of the data. Allowing SSTables to organically churn into 
the new version via compaction avoids this.If you're upgrading from 3.x to 4.x, don't feel like you 
have to - but it does avoid the need to run upgradesstables in a hypothetical distant future.– 
ScottOn Aug 16, 2022, at 6:32 AM, Jai Bheemsen Rao Dhanwada  
wrote:Thank you Erick,> it is going to be single-threaded by default so it will take a while to 
get through all the sstables on dense nodesIs there any downside if the upgradesstables take longer 
(example 1-2 days), other than I/O?Also when is the upgradesstable get triggered? after every node 
is upgraded or it will kick in only when all the nodes in the cluster upgraded to 4.0.x?On Tue, Aug 
16, 2022 at 2:12 AM Erick Ramirez  wrote:As convenient as it is, 
there are a few caveats and it isn't a silver bullet. The automatic feature will only kick in if 
there are no other compactions scheduled. Also, it is going to be single-threaded by default so it 
will take a while to get through all the sstables on dense nodes.In contrast, you'll have a bit more 
control if you manually upgrade the sstables. For example, you can schedule the upgrade during low 
traffic periods so reads are not competing with compactions for IO. Cheers!

Re: Cassandra 4.0 upgrade - Upgradesstables

[Jai Bheemsen Rao Dhanwada]

Thank you Erick,

> it is going to be single-threaded by default so it will take a while to
get through all the sstables on dense nodes
Is there any downside if the upgradesstables take longer (example 1-2
days), other than I/O?

Also when is the upgradesstable get triggered? after every node is
upgraded or it will kick in only when all the nodes in the cluster upgraded
to 4.0.x?

On Tue, Aug 16, 2022 at 2:12 AM Erick Ramirez 
wrote:

> As convenient as it is, there are a few caveats and it isn't a silver
> bullet. The automatic feature will only kick in if there are no other
> compactions scheduled. Also, it is going to be single-threaded by default
> so it will take a while to get through all the sstables on dense nodes.
>
> In contrast, you'll have a bit more control if you manually upgrade the
> sstables. For example, you can schedule the upgrade during low traffic
> periods so reads are not competing with compactions for IO. Cheers!
>
>>

Re: Cassandra 4.0 upgrade - Upgradesstables

[Erick Ramirez]

As convenient as it is, there are a few caveats and it isn't a silver
bullet. The automatic feature will only kick in if there are no other
compactions scheduled. Also, it is going to be single-threaded by default
so it will take a while to get through all the sstables on dense nodes.

In contrast, you'll have a bit more control if you manually upgrade the
sstables. For example, you can schedule the upgrade during low traffic
periods so reads are not competing with compactions for IO. Cheers!

>

Cassandra 4.0 upgrade - Upgradesstables

[Jai Bheemsen Rao Dhanwada]

Hello,

I am evaluating the upgrade from 3.11.x to 4.0.x and as per CASSANDRA-14197
 we don't need to
run upgradesstables any more. We have tested this in a test environment and
see that setting "-Dcassandra.automatic_sstable_upgrade=true" takes care of
the upgrading sstables to the new format. Since my actual cluster has a
very high data size and more datacenters, I would like to know the general
guidance on the upgradesstables option. Is this option guaranteed to take
care of upgradesstables or it's a best case scenario and expects the users
to run the upgradesstables (as I see some posts still run upgradesstables
for 4.x upgrade). If it's taken care of automatically, is there a way I can
see the progress/ time it takes or does the user need not worry about how
long it takes but can just proceed with the upgrade on all the data
centers sequentially one after another?

Thanks in advance.

Re: Cassandra 4.0 upgrade from Cassandra 3x

[manish khandelwal]

You can see upgrading instructions here
https://github.com/apache/cassandra/blob/cassandra-4.0.2/NEWS.txt.

On Fri, Feb 11, 2022 at 2:52 AM Abdul Patel  wrote:

> Hi
> apart from  standard upgrade process any thing specific needs ti be
> handled separately for this upgrade process?
>
> Any changes needed at client side w.r.t drivers?
>

Re: Cassandra 4.0 upgrade from Cassandra 3x

[Erick Ramirez]

Make sure you go through all the instructions in
https://github.com/apache/cassandra/blob/trunk/NEWS.txt. It's also highly
recommended that you upgrade to the latest 3.0.x or 3.11.x version before
upgrading to 4.0.

Generally there are no changes required on the client side apart from
setting the protocol version depending on the driver version. Cheers!

>

Cassandra 4.0 upgrade from Cassandra 3x

[Abdul Patel]

Hi
apart from  standard upgrade process any thing specific needs ti be handled
separately for this upgrade process?

Any changes needed at client side w.r.t drivers?