Re: Cassandra Upgrade with Different Protocol Version
> On Jul 5, 2018, at 12:45 PM, Anuj Wadehra > wrote: > > Hi, > > I woud like to know how people are doing rolling upgrade of Casandra clustes > when there is a change in native protocol version say from 2.1 to 3.11. > During rolling upgrade, if client application is restarted on nodes, the > client driver may first contact an upgraded Cassandra node with v4 and > permanently mark all old Casandra nodes on v3 as down. This may lead to > request failures. Datastax recommends two ways to deal with this: > > 1. Before upgrade, set protocol version to lower protocol version. And move > to higher version once entire cluster is upgraded. > 2. Make sure driver only contacts upraded Cassandra nodes during rolling > upgrade. 3. Make sure driver only contacts nonupgraded nodes during upgrade > > Second workaround will lead to failures as you may not be able to meet > required consistency for some time. That definitely shouldn’t be true. Querying a 3.x node will internally query the 2.1 nodes and translate it. The 3.0 node will talk to the 2.1 instances using the 2.1 internode messaging protocol. > > Lets consider first workaround. Now imagine an application where protocol > version is not configurable and code uses default protocol version. You can > not apply first workaroud because you have to upgrade your application on all > nodes to first make the protocol version configurable. How would you upgrade > such a cluster without downtime? Thoughts? You could try turning off native protocol on the upgraded hosts until you had enough to serve the load, then switch on native on the 3.0 hosts, let connections move, and then native off for 2.1 Alternatively, depending on your driver (and it’s discovery mechanism), you may be able to start some instances that only listen on the v3 protocol without owning any data (-Dcassandra.join_ring=false) and let clients connect there - then bounce through the cluster as needed. - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Cassandra Upgrade with Different Protocol Version
There is replication between 2.1 and 3.x, but not hints. You will have to repair past the window, but you should be doing that anyway if you care about tombstones doing the right thing Read quorum with 2/3 in either version should work fine - if it gives you an error please open a JIRA with steps to repro -- Jeff Jirsa > On Jul 5, 2018, at 6:39 PM, James Shaw wrote: > > other concerns: > there is no replication between 2.11 and 3.11, store in hints, and replay > hints when remote is same version. have to do repair if over window. if read > quorum 2/3, will get error. > > in case rollback to 2.11, can not read new version 3.11 data files, but > online rolling upgrade, some new data is in new version format. > > if hardlink snapshot is not copied to other device, then disk failure may > cause data loss. ( since may only have some data may just 1 copy during > upgrade because no replication). > > >> On Thu, Jul 5, 2018 at 8:13 PM, kooljava2 >> wrote: >> Hello Anuj, >> >> The 2nd workaround should work. As app will auto discover all the other >> nodes. Its the first contact with the node that app makes determines the >> protocol version. So if you remove the newer version nodes from the app >> configuration after the startup, it will auto discover the newer nodes as >> well. >> >> Thank you, >> TS. >> >> On Thursday, 5 July 2018, 12:45:39 GMT-7, Anuj Wadehra >> wrote: >> >> >> Hi, >> >> I woud like to know how people are doing rolling upgrade of Casandra clustes >> when there is a change in native protocol version say from 2.1 to 3.11. >> During rolling upgrade, if client application is restarted on nodes, the >> client driver may first contact an upgraded Cassandra node with v4 and >> permanently mark all old Casandra nodes on v3 as down. This may lead to >> request failures. Datastax recommends two ways to deal with this: >> >> 1. Before upgrade, set protocol version to lower protocol version. And move >> to higher version once entire cluster is upgraded. >> 2. Make sure driver only contacts upraded Cassandra nodes during rolling >> upgrade. >> >> Second workaround will lead to failures as you may not be able to meet >> required consistency for some time. >> >> Lets consider first workaround. Now imagine an application where protocol >> version is not configurable and code uses default protocol version. You can >> not apply first workaroud because you have to upgrade your application on >> all nodes to first make the protocol version configurable. How would you >> upgrade such a cluster without downtime? Thoughts? >> >> Thanks >> Anuj >> >> >
Re: Cassandra Upgrade with Different Protocol Version
other concerns: there is no replication between 2.11 and 3.11, store in hints, and replay hints when remote is same version. have to do repair if over window. if read quorum 2/3, will get error. in case rollback to 2.11, can not read new version 3.11 data files, but online rolling upgrade, some new data is in new version format. if hardlink snapshot is not copied to other device, then disk failure may cause data loss. ( since may only have some data may just 1 copy during upgrade because no replication). On Thu, Jul 5, 2018 at 8:13 PM, kooljava2 wrote: > Hello Anuj, > > The 2nd workaround should work. As app will auto discover all the other > nodes. Its the first contact with the node that app makes determines the > protocol version. So if you remove the newer version nodes from the app > configuration after the startup, it will auto discover the newer nodes as > well. > > Thank you, > TS. > > On Thursday, 5 July 2018, 12:45:39 GMT-7, Anuj Wadehra < > anujw_2...@yahoo.co.in.INVALID> wrote: > > > Hi, > > I woud like to know how people are doing rolling upgrade of Casandra > clustes when there is a change in native protocol version say from 2.1 to > 3.11. During rolling upgrade, if client application is restarted on nodes, > the client driver may first contact an upgraded Cassandra node with v4 and > permanently mark all old Casandra nodes on v3 as down. This may lead to > request failures. Datastax recommends two ways to deal with this: > > 1. Before upgrade, set protocol version to lower protocol version. And > move to higher version once entire cluster is upgraded. > 2. Make sure driver only contacts upraded Cassandra nodes during rolling > upgrade. > > Second workaround will lead to failures as you may not be able to meet > required consistency for some time. > > Lets consider first workaround. Now imagine an application where protocol > version is not configurable and code uses default protocol version. You can > not apply first workaroud because you have to upgrade your application on > all nodes to first make the protocol version configurable. How would you > upgrade such a cluster without downtime? Thoughts? > > Thanks > Anuj > > >
Re: Cassandra Upgrade with Different Protocol Version
Hello Anuj, The 2nd workaround should work. As app will auto discover all the other nodes. Its the first contact with the node that app makes determines the protocol version. So if you remove the newer version nodes from the app configuration after the startup, it will auto discover the newer nodes as well. Thank you,TS. On Thursday, 5 July 2018, 12:45:39 GMT-7, Anuj Wadehra wrote: Hi, I woud like to know how people are doing rolling upgrade of Casandra clustes when there is a change in native protocol version say from 2.1 to 3.11. During rolling upgrade, if client application is restarted on nodes, the client driver may first contact an upgraded Cassandra node with v4 and permanently mark all old Casandra nodes on v3 as down. This may lead to request failures. Datastax recommends two ways to deal with this: 1. Before upgrade, set protocol version to lower protocol version. And move to higher version once entire cluster is upgraded.2. Make sure driver only contacts upraded Cassandra nodes during rolling upgrade. Second workaround will lead to failures as you may not be able to meet required consistency for some time. Lets consider first workaround. Now imagine an application where protocol version is not configurable and code uses default protocol version. You can not apply first workaroud because you have to upgrade your application on all nodes to first make the protocol version configurable. How would you upgrade such a cluster without downtime? Thoughts? ThanksAnuj
Cassandra Upgrade with Different Protocol Version
Hi, I woud like to know how people are doing rolling upgrade of Casandra clustes when there is a change in native protocol version say from 2.1 to 3.11. During rolling upgrade, if client application is restarted on nodes, the client driver may first contact an upgraded Cassandra node with v4 and permanently mark all old Casandra nodes on v3 as down. This may lead to request failures. Datastax recommends two ways to deal with this: 1. Before upgrade, set protocol version to lower protocol version. And move to higher version once entire cluster is upgraded.2. Make sure driver only contacts upraded Cassandra nodes during rolling upgrade. Second workaround will lead to failures as you may not be able to meet required consistency for some time. Lets consider first workaround. Now imagine an application where protocol version is not configurable and code uses default protocol version. You can not apply first workaroud because you have to upgrade your application on all nodes to first make the protocol version configurable. How would you upgrade such a cluster without downtime? Thoughts? ThanksAnuj
