Re: [UPGRADATION] Apache Cassandra from version 3.0.9 to 4.0.0

2021-09-06 Thread MyWorld 
Thanks all for the help.

On Mon, Sep 6, 2021 at 2:20 PM manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> Totally agree with Jeff and Bowen there. Don't try to achieve something
> faster by cutting corners. Migration to GCP from physical DC should be done
> on the same versions.
>
> On Mon, Sep 6, 2021 at 2:11 PM Bowen Song  wrote:
>
>> Hello Ashish,
>>
>>
>> I'm slightly worried about this:
>>
>> *Since I won't be needing physical DC anymore so instead of upgrading it
>> I will simply discard that DC*
>>
>> This sounds like you are planning to add GCP 3.x to existing cluster, and
>> upgrade GCP to 4.0, then decommission the existing DC without upgrading. If
>> so, you need to think twice. Adding or removing nodes (or DCs) in a cluster
>> with different versions is not recommended. I'd highly recommend you
>> upgrade the existing DC before decommissioning it. Of course, you can skip
>> the upgrade sstables on it which is often the most time consuming part.
>>
>>
>> Cheers,
>>
>> Bowen
>>
>> On 06/09/2021 03:29, MyWorld wrote:
>>
>> Hi Jeff,
>>
>> *When you’re upgrading or rebuilding you want all copies on the same
>> version with proper sstables . So either add GCP then upgrade to 4.0 or
>> upgrade to 4.0 and then expand to GCP. Don’t do them at the same time.*
>>
>> I think I forgot to mention one thing that after completion of step 1 our
>> GCP data center will be added with rebuild done on all nodes. So our
>> complete cluster would be on 3.0.9 after step 1. Will change num_tokens
>> from current 256 to 16 in GCP data center in this step only.
>>
>> DC1 -
>> 5nodes (physical) - version 3.0.9
>> numtokens256
>> DC2 -
>> 5nodes (GCP) - version 3.0.9
>> numtokens16
>>
>> Rest all step from 2-5 are meant for upgradation in which I am planning
>> to go DC wise upgradation and running upgradesstables on GCP first.
>>
>> DC1 -
>> 5nodes (physical) - version 3.0.9
>> numtokens256
>> DC2 -
>> 5nodes (GCP) - version 4.0.0
>> numtokens16
>>
>> Since I won't be needing physical DC anymore so instead of upgrading it I
>> will simply discard that DC
>>
>> Regards,
>> Ashish
>>
>> On Mon, Sep 6, 2021, 7:31 AM Jeff Jirsa  wrote:
>>
>>> In-line
>>>
>>> On Sep 3, 2021, at 11:12 AM, MyWorld  wrote:
>>>
>>> 
>>> Hi Jeff,
>>> Thanks for your response.
>>> To answer your question, Yes, we have created dev environment by
>>> restoring them from snapshot/CSV files.
>>>
>>> Just one follow up question, I have a 5-node single DC on production on
>>> version 3.0.9on physical server.
>>> We are planning to migrate to GCP along with upgradation using below
>>> steps.
>>> 1. Setup GCP data center with same version 3.0.9 and rebuild complete
>>> data
>>> 2. Now install and configure 4.0 version in new GCP data center on all 5
>>> nodes
>>> 3. Stop version 3.0.9 and start 4.0 on all 5 nodes of GCP one by one
>>> 4. Run upgradesstables one by one on all 5 nodes of GCP
>>> 5.Later move read/write traffic to GCP and remove old datacenter which
>>> is still on version 3.0.9
>>>
>>> Please guide on few things:
>>> 1. Is the above mention approach right?
>>>
>>>
>>> When you’re upgrading or rebuilding you want all copies on the same
>>> version with proper sstables . So either add GCP then upgrade to 4.0 or
>>> upgrade to 4.0 and then expand to GCP. Don’t do them at the same time.
>>>
>>>
>>> 2. OR should we update 4.0 on only one node on GCP at a time and run
>>> upgrade sstables on just one node first
>>>
>>>
>>> I usually do upgradesstables after all bounces are done
>>>
>>> The only exception is perhaps doing upgradesstables with exactly one
>>> copy via backup/restore to make sure 4.0 works with your data files, which
>>> it sounds like you’ve already done.
>>>
>>> 3. OR should we migrate to GCP first and then think of upgrade 4.0 later
>>> 4. OR Is there any reason I should upgrade to 3.11.x first
>>>
>>>
>>> Not 3.11 but maybe latest 3.0 instead
>>>
>>>
>>>
>>> Regards,
>>> Ashish
>>>
>>> On Fri, Sep 3, 2021, 11:11 PM Jeff Jirsa  wrote:
>>>


 On Fri, Sep 3, 2021 at 10:33 AM MyWorld 
 wrote:

> Hi all,
> We are doing a POC on dev environment to upgrade apache cassandra
> 3.0.9 to 4.0.0. We have the below setup currently on cassandra 3.0.9
> DC1 - GCP(india) - 1 node
> DC2 - GCP(US) - 1 node
>

 3.0.9 is very old. It's got older version of data files and some known
 correctness bugs.


>
> For upgradation, we carried out below steps on DC2 - GCP(US) node:
> Step1. Install apache cassandra 4.0.0
> Step2. Did all Configuration settings
> Step3. Stop apache cassandra 3.0.9
> Step4. Start apache cassandra 4.0.0 and monitor logs
> Step5. Run nodetool upgradesstables and monitor logs
>
> After monitoring logs, I had below observations:
> *1. Initially during bootstrap at Step4, received below exceptions:*
> a) Exception (java.lang.IllegalArgumentException) encountered during
> startup: Invalid sstable file

Re: [UPGRADATION] Apache Cassandra from version 3.0.9 to 4.0.0

2021-09-06 Thread manish khandelwal 
Totally agree with Jeff and Bowen there. Don't try to achieve something
faster by cutting corners. Migration to GCP from physical DC should be done
on the same versions.

On Mon, Sep 6, 2021 at 2:11 PM Bowen Song  wrote:

> Hello Ashish,
>
>
> I'm slightly worried about this:
>
> *Since I won't be needing physical DC anymore so instead of upgrading it I
> will simply discard that DC*
>
> This sounds like you are planning to add GCP 3.x to existing cluster, and
> upgrade GCP to 4.0, then decommission the existing DC without upgrading. If
> so, you need to think twice. Adding or removing nodes (or DCs) in a cluster
> with different versions is not recommended. I'd highly recommend you
> upgrade the existing DC before decommissioning it. Of course, you can skip
> the upgrade sstables on it which is often the most time consuming part.
>
>
> Cheers,
>
> Bowen
>
> On 06/09/2021 03:29, MyWorld wrote:
>
> Hi Jeff,
>
> *When you’re upgrading or rebuilding you want all copies on the same
> version with proper sstables . So either add GCP then upgrade to 4.0 or
> upgrade to 4.0 and then expand to GCP. Don’t do them at the same time.*
>
> I think I forgot to mention one thing that after completion of step 1 our
> GCP data center will be added with rebuild done on all nodes. So our
> complete cluster would be on 3.0.9 after step 1. Will change num_tokens
> from current 256 to 16 in GCP data center in this step only.
>
> DC1 -
> 5nodes (physical) - version 3.0.9
> numtokens256
> DC2 -
> 5nodes (GCP) - version 3.0.9
> numtokens16
>
> Rest all step from 2-5 are meant for upgradation in which I am planning to
> go DC wise upgradation and running upgradesstables on GCP first.
>
> DC1 -
> 5nodes (physical) - version 3.0.9
> numtokens256
> DC2 -
> 5nodes (GCP) - version 4.0.0
> numtokens16
>
> Since I won't be needing physical DC anymore so instead of upgrading it I
> will simply discard that DC
>
> Regards,
> Ashish
>
> On Mon, Sep 6, 2021, 7:31 AM Jeff Jirsa  wrote:
>
>> In-line
>>
>> On Sep 3, 2021, at 11:12 AM, MyWorld  wrote:
>>
>> 
>> Hi Jeff,
>> Thanks for your response.
>> To answer your question, Yes, we have created dev environment by
>> restoring them from snapshot/CSV files.
>>
>> Just one follow up question, I have a 5-node single DC on production on
>> version 3.0.9on physical server.
>> We are planning to migrate to GCP along with upgradation using below
>> steps.
>> 1. Setup GCP data center with same version 3.0.9 and rebuild complete data
>> 2. Now install and configure 4.0 version in new GCP data center on all 5
>> nodes
>> 3. Stop version 3.0.9 and start 4.0 on all 5 nodes of GCP one by one
>> 4. Run upgradesstables one by one on all 5 nodes of GCP
>> 5.Later move read/write traffic to GCP and remove old datacenter which is
>> still on version 3.0.9
>>
>> Please guide on few things:
>> 1. Is the above mention approach right?
>>
>>
>> When you’re upgrading or rebuilding you want all copies on the same
>> version with proper sstables . So either add GCP then upgrade to 4.0 or
>> upgrade to 4.0 and then expand to GCP. Don’t do them at the same time.
>>
>>
>> 2. OR should we update 4.0 on only one node on GCP at a time and run
>> upgrade sstables on just one node first
>>
>>
>> I usually do upgradesstables after all bounces are done
>>
>> The only exception is perhaps doing upgradesstables with exactly one copy
>> via backup/restore to make sure 4.0 works with your data files, which it
>> sounds like you’ve already done.
>>
>> 3. OR should we migrate to GCP first and then think of upgrade 4.0 later
>> 4. OR Is there any reason I should upgrade to 3.11.x first
>>
>>
>> Not 3.11 but maybe latest 3.0 instead
>>
>>
>>
>> Regards,
>> Ashish
>>
>> On Fri, Sep 3, 2021, 11:11 PM Jeff Jirsa  wrote:
>>
>>>
>>>
>>> On Fri, Sep 3, 2021 at 10:33 AM MyWorld  wrote:
>>>
 Hi all,
 We are doing a POC on dev environment to upgrade apache cassandra 3.0.9
 to 4.0.0. We have the below setup currently on cassandra 3.0.9
 DC1 - GCP(india) - 1 node
 DC2 - GCP(US) - 1 node

>>>
>>> 3.0.9 is very old. It's got older version of data files and some known
>>> correctness bugs.
>>>
>>>

 For upgradation, we carried out below steps on DC2 - GCP(US) node:
 Step1. Install apache cassandra 4.0.0
 Step2. Did all Configuration settings
 Step3. Stop apache cassandra 3.0.9
 Step4. Start apache cassandra 4.0.0 and monitor logs
 Step5. Run nodetool upgradesstables and monitor logs

 After monitoring logs, I had below observations:
 *1. Initially during bootstrap at Step4, received below exceptions:*
 a) Exception (java.lang.IllegalArgumentException) encountered during
 startup: Invalid sstable file manifest.json: the name doesn't look like a
 supported sstable file name
 java.lang.IllegalArgumentException: Invalid sstable file manifest.json:
 the name doesn't look like a supported sstable file name
 b) ERROR [main] 2021-08-29 06:25:52,120

Re: [UPGRADATION] Apache Cassandra from version 3.0.9 to 4.0.0

2021-09-06 Thread Bowen Song 

Hello Ashish,


I'm slightly worried about this:

   /Since I won't be needing physical DC anymore so instead of
   upgrading it I will simply discard that DC/

This sounds like you are planning to add GCP 3.x to existing cluster, 
and upgrade GCP to 4.0, then decommission the existing DC without 
upgrading. If so, you need to think twice. Adding or removing nodes (or 
DCs) in a cluster with different versions is not recommended. I'd highly 
recommend you upgrade the existing DC before decommissioning it. Of 
course, you can skip the upgrade sstables on it which is often the most 
time consuming part.



Cheers,

Bowen


On 06/09/2021 03:29, MyWorld wrote:

Hi Jeff,

/When you’re upgrading or rebuilding you want all copies on the same 
version with proper sstables . So either add GCP then upgrade to 4.0 
or upgrade to 4.0 and then expand to GCP. Don’t do them at the same time./


I think I forgot to mention one thing that after completion of step 1 
our GCP data center will be added with rebuild done on all nodes. So 
our complete cluster would be on 3.0.9 after step 1. Will change 
num_tokens from current 256 to 16 in GCP data center in this step only.


DC1 -
5nodes (physical) - version 3.0.9
numtokens256
DC2 -
5nodes (GCP) - version 3.0.9
numtokens16

Rest all step from 2-5 are meant for upgradation in which I am 
planning to go DC wise upgradation and running upgradesstables on GCP 
first.


DC1 -
5nodes (physical) - version 3.0.9
numtokens256
DC2 -
5nodes (GCP) - version 4.0.0
numtokens16

Since I won't be needing physical DC anymore so instead of upgrading 
it I will simply discard that DC


Regards,
Ashish

On Mon, Sep 6, 2021, 7:31 AM Jeff Jirsa > wrote:


In-line


On Sep 3, 2021, at 11:12 AM, MyWorld mailto:timeplus.1...@gmail.com>> wrote:


Hi Jeff,
Thanks for your response.
To answer your question, Yes, we have created dev environment by
restoring them from snapshot/CSV files.

Just one follow up question, I have a 5-node single DC on
production on version 3.0.9on physical server.
We are planning to migrate to GCP along with upgradation using
below steps.
1. Setup GCP data center with same version 3.0.9 and rebuild
complete data
2. Now install and configure 4.0 version in new GCP data center
on all 5 nodes
3. Stop version 3.0.9 and start 4.0 on all 5 nodes of GCP one by one
4. Run upgradesstables one by one on all 5 nodes of GCP
5.Later move read/write traffic to GCP and remove old datacenter
which is still on version 3.0.9

Please guide on few things:
1. Is the above mention approach right?


When you’re upgrading or rebuilding you want all copies on the
same version with proper sstables . So either add GCP then upgrade
to 4.0 or upgrade to 4.0 and then expand to GCP. Don’t do them at
the same time.



2. OR should we update 4.0 on only one node on GCP at a time and
run upgrade sstables on just one node first


I usually do upgradesstables after all bounces are done

The only exception is perhaps doing upgradesstables with exactly
one copy via backup/restore to make sure 4.0 works with your data
files, which it sounds like you’ve already done.


3. OR should we migrate to GCP first and then think of upgrade
4.0 later
4. OR Is there any reason I should upgrade to 3.11.x first


Not 3.11 but maybe latest 3.0 instead




Regards,
Ashish

On Fri, Sep 3, 2021, 11:11 PM Jeff Jirsa mailto:jji...@gmail.com>> wrote:



On Fri, Sep 3, 2021 at 10:33 AM MyWorld
mailto:timeplus.1...@gmail.com>> wrote:

Hi all,
We are doing a POC on dev environment to upgrade apache
cassandra 3.0.9 to 4.0.0. We have the below setup
currently on cassandra 3.0.9
DC1 - GCP(india) - 1 node
DC2 - GCP(US) - 1 node


3.0.9 is very old. It's got older version of data files and
some known correctness bugs.


For upgradation, we carried out below steps on DC2 -
GCP(US) node:
Step1. Install apache cassandra 4.0.0
Step2. Did all Configuration settings
Step3. Stop apache cassandra 3.0.9
Step4. Start apache cassandra 4.0.0 and monitor logs
Step5. Run nodetool upgradesstables and monitor logs

After monitoring logs, I had below observations:
*1. Initially during bootstrap at Step4, received below
exceptions:*
a) Exception (java.lang.IllegalArgumentException)
encountered during startup: Invalid sstable file
manifest.json: the name doesn't look like a supported
sstable file name
java.lang.IllegalArgumentException: Invalid sstable file
manifest.json: the name doesn't look like a supported
sstable file name
b) ERROR [main] 2021-08-29 06:25:52,120