Re: Upgrading existing non-TLS cluster with no downtime
Thanks Mate for the responses. Brings a lot of clarity to me. I was able to get it working this time without downtime. Don't know what I did wrong the last time though. On Mon, 20 Jul 2020 at 12:57, Szalay-Bekő Máté wrote: > echo "stat" | nc localhost 2182Hi, > > I guess this is the part you are referring: > > https://zookeeper.apache.org/doc/r3.5.8/zookeeperAdmin.html#Upgrading+existing+nonTLS+cluster > (your link was pointing to the 3.3.2 admin guide where this chapter was > missing) > > > 1) When I set sslQuorum=true and portUnification=true on the first > server, > does it go out of the quorum? And when these properties are set in the > second server, a new quorum of first and second server is formed and now > the third server is out of quorum. When the 3rd server follows suit, it is > added back to the quorum. > > the "sslQuorum=true and portUnification=true" setting is needed in step 4 > (although the numbering is bad in the markdown...). After step 3 you > already have a 3 server quorum up with portUnification=true, meaning the > cluster can handle both TLS/SSL and regular/non-secure connections. So when > you restart server 1 with sslQuorum=true, then it will be able to re-join > to the quorum, as server 2 and 3 are capable of handling SSL connections > (even if they are not using it for connection initiation). So ideally > between restarting each servers with sslQuorum=true, you always should have > a 3 node full quorum. > > > 2) The guideline says to check after restarting every broker that the > quorum is healthy, is there any metric to track that? > > I send the "stat" command to all nodes to see if everyone is connected to > the quorum. E.g.: echo "stat" | nc localhost 2181 > I usually use 4-letter-word commands but the REST admin API works as well, > and actually that is the officially recommended way, as the 4-letter-words > are / will be deprecated some time. > For the admin server see: > https://zookeeper.apache.org/doc/r3.5.8/zookeeperAdmin.html#sc_adminserver > > Kind regards, > Mate > > On Tue, Jul 14, 2020 at 10:52 PM Sankalp Bhatia > > wrote: > > > +users > > > > On Tue, 14 Jul 2020 at 21:51, Sankalp Bhatia > > wrote: > > > > > Hi All, > > > > > > I am trying to follow the section "Upgrading existing non-TLS cluster > > with > > > no downtime" in the zookeeper guide : > > > https://zookeeper.apache.org/doc/r3.3.2/zookeeperAdmin.html > > > > > > I have an ensemble of 3 servers. I have a couple of questions: > > > > > > 1) When I set sslQuorum=true and portUnification=true on the first > > > server, does it go out of the quorum? And when these properties are set > > > in the second server, a new quorum of first and second server is formed > > and > > > now the third server is out of quorum. When the 3rd server follows > suit, > > it > > > is added back to the quorum. > > > > > > If this is the case, what is the use of a the port-unification feature > > > here? > > > > > > 2) The guideline says to check after restarting every broker that the > > > quorum is healthy, is there any metric to track that? > > > > > > Thanks, > > > Sankalp > > > > > > > > > > > > > > >
Re: Upgrading existing non-TLS cluster with no downtime
echo "stat" | nc localhost 2182Hi, I guess this is the part you are referring: https://zookeeper.apache.org/doc/r3.5.8/zookeeperAdmin.html#Upgrading+existing+nonTLS+cluster (your link was pointing to the 3.3.2 admin guide where this chapter was missing) > 1) When I set sslQuorum=true and portUnification=true on the first server, does it go out of the quorum? And when these properties are set in the second server, a new quorum of first and second server is formed and now the third server is out of quorum. When the 3rd server follows suit, it is added back to the quorum. the "sslQuorum=true and portUnification=true" setting is needed in step 4 (although the numbering is bad in the markdown...). After step 3 you already have a 3 server quorum up with portUnification=true, meaning the cluster can handle both TLS/SSL and regular/non-secure connections. So when you restart server 1 with sslQuorum=true, then it will be able to re-join to the quorum, as server 2 and 3 are capable of handling SSL connections (even if they are not using it for connection initiation). So ideally between restarting each servers with sslQuorum=true, you always should have a 3 node full quorum. > 2) The guideline says to check after restarting every broker that the quorum is healthy, is there any metric to track that? I send the "stat" command to all nodes to see if everyone is connected to the quorum. E.g.: echo "stat" | nc localhost 2181 I usually use 4-letter-word commands but the REST admin API works as well, and actually that is the officially recommended way, as the 4-letter-words are / will be deprecated some time. For the admin server see: https://zookeeper.apache.org/doc/r3.5.8/zookeeperAdmin.html#sc_adminserver Kind regards, Mate On Tue, Jul 14, 2020 at 10:52 PM Sankalp Bhatia wrote: > +users > > On Tue, 14 Jul 2020 at 21:51, Sankalp Bhatia > wrote: > > > Hi All, > > > > I am trying to follow the section "Upgrading existing non-TLS cluster > with > > no downtime" in the zookeeper guide : > > https://zookeeper.apache.org/doc/r3.3.2/zookeeperAdmin.html > > > > I have an ensemble of 3 servers. I have a couple of questions: > > > > 1) When I set sslQuorum=true and portUnification=true on the first > > server, does it go out of the quorum? And when these properties are set > > in the second server, a new quorum of first and second server is formed > and > > now the third server is out of quorum. When the 3rd server follows suit, > it > > is added back to the quorum. > > > > If this is the case, what is the use of a the port-unification feature > > here? > > > > 2) The guideline says to check after restarting every broker that the > > quorum is healthy, is there any metric to track that? > > > > Thanks, > > Sankalp > > > > > > > > >
Re: Upgrading existing non-TLS cluster with no downtime
+users On Tue, 14 Jul 2020 at 21:51, Sankalp Bhatia wrote: > Hi All, > > I am trying to follow the section "Upgrading existing non-TLS cluster with > no downtime" in the zookeeper guide : > https://zookeeper.apache.org/doc/r3.3.2/zookeeperAdmin.html > > I have an ensemble of 3 servers. I have a couple of questions: > > 1) When I set sslQuorum=true and portUnification=true on the first > server, does it go out of the quorum? And when these properties are set > in the second server, a new quorum of first and second server is formed and > now the third server is out of quorum. When the 3rd server follows suit, it > is added back to the quorum. > > If this is the case, what is the use of a the port-unification feature > here? > > 2) The guideline says to check after restarting every broker that the > quorum is healthy, is there any metric to track that? > > Thanks, > Sankalp > > > >