Thanks Sean, In my use case all my clusters are multi DC, and I am trying my best effort to upgrade ASAP, however there is a chance since all machines are VMs. Also my key spaces are not uniform across DCs. some are replicated to all DCs and some of them are just one DC, so I am worried there.
Is there a way to override the protocol version until the upgrade is done and then change it back once the upgrade is completed? On Fri, Jul 26, 2019 at 11:42 AM Durity, Sean R <sean_r_dur...@homedepot.com> wrote: > What you have seen is totally expected. You can’t stream between different > major versions of Cassandra. Get the upgrade done, then worry about any > down hardware. If you are using DCs, upgrade one DC at a time, so that > there is an available environment in case of any disasters. > > > > My advice, though, is to get through the rolling upgrade process as > quickly as possible. Don’t stay in a mixed state very long. The cluster > will function fine in a mixed state – except for those streaming > operations. No repairs, no bootstraps. > > > > > > Sean Durity – Staff Systems Engineer, Cassandra > > > > *From:* Jai Bheemsen Rao Dhanwada <jaibheem...@gmail.com> > *Sent:* Friday, July 26, 2019 2:24 PM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] Apache Cassandra upgrade path > > > > Hello, > > > > I am trying to upgrade Apache Cassandra from 2.1.16 to 3.11.3, the regular > rolling upgrade process works fine without any issues. > > > > However, I am running into an issue where if there is a node with older > version dies (hardware failure) and a new node comes up and tries to > bootstrap, it's failing. > > > > I tried two combinations: > > > > 1. Joining replacement node with 2.1.16 version of cassandra > > In this case nodes with 2.1.16 version are able to stream data to the new > node, but the nodes with 3.11.3 version are failing with the below error. > > > > ERROR [STREAM-INIT-/10.x.x.x:40296] 2019-07-26 17:45:17,775 > IncomingStreamingConnection.java:80 - Error while reading from socket from > /10.y.y.y:40296. > java.io.IOException: Received stream using protocol version 2 (my version > 4). Terminating connection > > 2. Joining replacement node with 3.11.3 version of cassandra > > In this case the nodes with 3.11.3 version of cassandra are able to stream > the data but it's not able to stream data from the 2.1.16 nodes and failing > with the below error. > > > > ERROR [STREAM-IN-/10.z.z.z:7000] 2019-07-26 18:08:10,380 > StreamSession.java:593 - [Stream #538c6900-afd0-11e9-a649-ab2e045ee53b] > Streaming error occurred on session with peer 10.z.z.z > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > ~[na:1.8.0_151] > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > ~[na:1.8.0_151] > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > ~[na:1.8.0_151] > at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_151] > at > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) > ~[na:1.8.0_151] > at > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206) > ~[na:1.8.0_151] > at > sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > ~[na:1.8.0_151] > at > java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) > ~[na:1.8.0_151] > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:311) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] > > > > Note: In both cases I am using replace_address to replace dead node, as I > am running into some issues with "nodetool removenode" . I use ephemeral > disk, so replacement node always comes up with empty data dir and bootstrap. > > > > Any other work around to mitigate this problem? I am worried about any > nodes going down while we are in the process of upgrade, as it could take > several hours to upgrade depending on the cluster size. > > ------------------------------ > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to the terms and conditions expressed in any applicable governing The Home > Depot terms of business or client engagement letter. The Home Depot > disclaims all responsibility and liability for the accuracy and content of > this attachment and for any damages or losses arising from any > inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other > items of a destructive nature, which may be contained in this attachment > and shall not be liable for direct, indirect, consequential or special > damages in connection with this e-mail message or its attachment. >
