Re: Observer properties for SASL authentication in 3.4.13 version
Hi, After a long time i have tried this again, i have removed the observer type but still failing. Ram On Sat, Sep 29, 2018 at 11:50 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > I will try number 1 and yes there is no such entry in host file. > > On Sat, Sep 29, 2018, 10:37 AM Rakesh Radhakrishnan > wrote: > >> OK, it looks to me some common networking related issue. >> >> 1) To confirm, can you remove the Observer type and simply try to join zk >> server to quorum like participant? >> >> 2) Can you also confirm, hope you don't have "hostname" from the 127.0.0.1 >> line in /etc/hosts. Something like, >> >>127.0.0.1 node203ea localhost localhost.localdomain >> localhost4 >> localhost4.localdomain4 >>::1 localhost localhost.localdomain localhost6 >> localhost6.localdomain6 >> >> http://ccl.cse.nd.edu/operations/condor/hostname.shtml >> >> On Fri, Sep 28, 2018 at 10:25 PM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >> > Any thoughts on what could be the reason for observers not able to >> connect >> > to followers/leader? >> > >> > Ram >> > >> > On Thu, Sep 27, 2018 at 1:00 PM rammohan ganapavarapu < >> > rammohanga...@gmail.com> wrote: >> > >> >> Incase if you have not received my previous logs files. >> >> >> >> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu < >> >> rammohanga...@gmail.com> wrote: >> >> >> >>> Rakesh, >> >>> >> >>> Thank you, i have 3 floower and 3 observers in two different DC's >> >>> followers came up fine with SASL but for some reasons observers are >> not >> >>> coming up with the following error but i dont see any network issues, >> i was >> >>> able to telnet to 2181 and 3888 ports. >> >>> >> >>> >> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - >> Queue >> >>> size: 1 >> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - >> Queue >> >>> size: 1 >> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - >> Queue >> >>> size: 1 >> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - >> >>> Opening channel to server 1 >> >>> 2018-09-24 17:55:34,151 [myid:6] - WARN >> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - >> Cannot >> >>> open channel to 1 at election address zk-server1/10.16.1.102:3888 >> >>> java.net.SocketTimeoutException: connect timed out >> >>> at java.net.PlainSocketImpl.socketConnect(Native Method) >> >>> at >> >>> java.net >> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >> >>> at >> >>> java.net >> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >> >>> at >> >>> java.net >> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >> >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >> >>> at java.net.Socket.connect(Socket.java:589) >> >>> at >> >>> >> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) >> >>> at >> >>> >> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) >> >>> at >> >>> >> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) >> >>> at >> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) >> >>> >> >>> >> >>> server.1=zk-server1:2888:3888 >> >>> server.2=zk-server2:2888:3888 >> >>> server.3=zk-server3:2888:3888 >> >>> server.4=zk-server4:2888:3888:observer >> >>> server.5=zk-server5:2888:3888:observer >> >>> server.6=zk-server6:2888:3888:observer >> >>> peerType=observer >> >>> >> >>> What could be the reason? >> >>> >> >>> Ram >> >>> >> >>> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan < >> >>> rake...@apache.org> wrote: >> >>> >> Thanks Ram for the interest on this feature. >> >> Yes, user can enable SASL for Observer nodes as well. In general, >> QuorumLearner will send authentication packet to peer QuorumServer. >> Observer is a learner which follows the same quorum authentication >> protocol >> and auth logic will work fine. >> >> FYI, hope you are referring below links for configurations, >> >> >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication >> >> >> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ >> >> Please let us know if you are facing any issues. >> >> Thanks, >> Rakesh >> >> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >> > Hi, >> > >> > Do we need to configure any thing on observer nodes for SASL >> > authentication? >> > >> > tcpKeepAlive=true ( this is
Re: Observer properties for SASL authentication in 3.4.13 version
I will try number 1 and yes there is no such entry in host file. On Sat, Sep 29, 2018, 10:37 AM Rakesh Radhakrishnan wrote: > OK, it looks to me some common networking related issue. > > 1) To confirm, can you remove the Observer type and simply try to join zk > server to quorum like participant? > > 2) Can you also confirm, hope you don't have "hostname" from the 127.0.0.1 > line in /etc/hosts. Something like, > >127.0.0.1 node203ea localhost localhost.localdomain localhost4 > localhost4.localdomain4 >::1 localhost localhost.localdomain localhost6 > localhost6.localdomain6 > > http://ccl.cse.nd.edu/operations/condor/hostname.shtml > > On Fri, Sep 28, 2018 at 10:25 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > > > Any thoughts on what could be the reason for observers not able to > connect > > to followers/leader? > > > > Ram > > > > On Thu, Sep 27, 2018 at 1:00 PM rammohan ganapavarapu < > > rammohanga...@gmail.com> wrote: > > > >> Incase if you have not received my previous logs files. > >> > >> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu < > >> rammohanga...@gmail.com> wrote: > >> > >>> Rakesh, > >>> > >>> Thank you, i have 3 floower and 3 observers in two different DC's > >>> followers came up fine with SASL but for some reasons observers are not > >>> coming up with the following error but i dont see any network issues, > i was > >>> able to telnet to 2181 and 3888 ports. > >>> > >>> > >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG > >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue > >>> size: 1 > >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG > >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue > >>> size: 1 > >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG > >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue > >>> size: 1 > >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG > >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - > >>> Opening channel to server 1 > >>> 2018-09-24 17:55:34,151 [myid:6] - WARN > >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - > Cannot > >>> open channel to 1 at election address zk-server1/10.16.1.102:3888 > >>> java.net.SocketTimeoutException: connect timed out > >>> at java.net.PlainSocketImpl.socketConnect(Native Method) > >>> at > >>> java.net > .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) > >>> at > >>> java.net > .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > >>> at > >>> java.net > .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > >>> at java.net.Socket.connect(Socket.java:589) > >>> at > >>> > org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) > >>> at > >>> > org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) > >>> at > >>> > org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) > >>> at > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) > >>> > >>> > >>> server.1=zk-server1:2888:3888 > >>> server.2=zk-server2:2888:3888 > >>> server.3=zk-server3:2888:3888 > >>> server.4=zk-server4:2888:3888:observer > >>> server.5=zk-server5:2888:3888:observer > >>> server.6=zk-server6:2888:3888:observer > >>> peerType=observer > >>> > >>> What could be the reason? > >>> > >>> Ram > >>> > >>> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan < > >>> rake...@apache.org> wrote: > >>> > Thanks Ram for the interest on this feature. > > Yes, user can enable SASL for Observer nodes as well. In general, > QuorumLearner will send authentication packet to peer QuorumServer. > Observer is a learner which follows the same quorum authentication > protocol > and auth logic will work fine. > > FYI, hope you are referring below links for configurations, > > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication > > > https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ > > Please let us know if you are facing any issues. > > Thanks, > Rakesh > > On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > > > Hi, > > > > Do we need to configure any thing on observer nodes for SASL > > authentication? > > > > tcpKeepAlive=true ( this is not for sasl but just asking ) > > > > quorum.auth.enableSasl=true > > quorum.auth.learnerRequireSasl=true > > quorum.auth.serverRequireSasl=true > > > > What will happen if i set these properties on observers nodes as > well ? > > > > Thanks, > > Ram > > > >
Re: Observer properties for SASL authentication in 3.4.13 version
OK, it looks to me some common networking related issue. 1) To confirm, can you remove the Observer type and simply try to join zk server to quorum like participant? 2) Can you also confirm, hope you don't have "hostname" from the 127.0.0.1 line in /etc/hosts. Something like, 127.0.0.1 node203ea localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 http://ccl.cse.nd.edu/operations/condor/hostname.shtml On Fri, Sep 28, 2018 at 10:25 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Any thoughts on what could be the reason for observers not able to connect > to followers/leader? > > Ram > > On Thu, Sep 27, 2018 at 1:00 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> Incase if you have not received my previous logs files. >> >> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> Rakesh, >>> >>> Thank you, i have 3 floower and 3 observers in two different DC's >>> followers came up fine with SASL but for some reasons observers are not >>> coming up with the following error but i dont see any network issues, i was >>> able to telnet to 2181 and 3888 ports. >>> >>> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >>> size: 1 >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >>> size: 1 >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >>> size: 1 >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - >>> Opening channel to server 1 >>> 2018-09-24 17:55:34,151 [myid:6] - WARN >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot >>> open channel to 1 at election address zk-server1/10.16.1.102:3888 >>> java.net.SocketTimeoutException: connect timed out >>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>> at >>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >>> at >>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >>> at >>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >>> at java.net.Socket.connect(Socket.java:589) >>> at >>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) >>> at >>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) >>> at >>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) >>> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) >>> >>> >>> server.1=zk-server1:2888:3888 >>> server.2=zk-server2:2888:3888 >>> server.3=zk-server3:2888:3888 >>> server.4=zk-server4:2888:3888:observer >>> server.5=zk-server5:2888:3888:observer >>> server.6=zk-server6:2888:3888:observer >>> peerType=observer >>> >>> What could be the reason? >>> >>> Ram >>> >>> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan < >>> rake...@apache.org> wrote: >>> Thanks Ram for the interest on this feature. Yes, user can enable SASL for Observer nodes as well. In general, QuorumLearner will send authentication packet to peer QuorumServer. Observer is a learner which follows the same quorum authentication protocol and auth logic will work fine. FYI, hope you are referring below links for configurations, https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ Please let us know if you are facing any issues. Thanks, Rakesh On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Hi, > > Do we need to configure any thing on observer nodes for SASL > authentication? > > tcpKeepAlive=true ( this is not for sasl but just asking ) > > quorum.auth.enableSasl=true > quorum.auth.learnerRequireSasl=true > quorum.auth.serverRequireSasl=true > > What will happen if i set these properties on observers nodes as well ? > > Thanks, > Ram >
Re: Observer properties for SASL authentication in 3.4.13 version
Any thoughts on what could be the reason for observers not able to connect to followers/leader? Ram On Thu, Sep 27, 2018 at 1:00 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Incase if you have not received my previous logs files. > > On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> Rakesh, >> >> Thank you, i have 3 floower and 3 observers in two different DC's >> followers came up fine with SASL but for some reasons observers are not >> coming up with the following error but i dont see any network issues, i was >> able to telnet to 2181 and 3888 ports. >> >> >> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >> size: 1 >> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >> size: 1 >> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >> size: 1 >> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - Opening >> channel to server 1 >> 2018-09-24 17:55:34,151 [myid:6] - WARN >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot >> open channel to 1 at election address zk-server1/10.16.1.102:3888 >> java.net.SocketTimeoutException: connect timed out >> at java.net.PlainSocketImpl.socketConnect(Native Method) >> at >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >> at >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >> at >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >> at java.net.Socket.connect(Socket.java:589) >> at >> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) >> at >> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) >> at >> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) >> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) >> >> >> server.1=zk-server1:2888:3888 >> server.2=zk-server2:2888:3888 >> server.3=zk-server3:2888:3888 >> server.4=zk-server4:2888:3888:observer >> server.5=zk-server5:2888:3888:observer >> server.6=zk-server6:2888:3888:observer >> peerType=observer >> >> What could be the reason? >> >> Ram >> >> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan >> wrote: >> >>> Thanks Ram for the interest on this feature. >>> >>> Yes, user can enable SASL for Observer nodes as well. In general, >>> QuorumLearner will send authentication packet to peer QuorumServer. >>> Observer is a learner which follows the same quorum authentication protocol >>> and auth logic will work fine. >>> >>> FYI, hope you are referring below links for configurations, >>> >>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication >>> >>> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ >>> >>> Please let us know if you are facing any issues. >>> >>> Thanks, >>> Rakesh >>> >>> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < >>> rammohanga...@gmail.com> wrote: >>> Hi, Do we need to configure any thing on observer nodes for SASL authentication? tcpKeepAlive=true ( this is not for sasl but just asking ) quorum.auth.enableSasl=true quorum.auth.learnerRequireSasl=true quorum.auth.serverRequireSasl=true What will happen if i set these properties on observers nodes as well ? Thanks, Ram >>>
Re: Observer properties for SASL authentication in 3.4.13 version
I'm in IST time zone and causes the delay:-) Have you verified zk cluster by not configuring "sasl" in all these servers and started, just to rule out the possibility of any errors with quorum authentication logic? Could you give more details: 1) Are you seeing that all Observers(4,5,6) are not able to connect to any of the quorum 1,2,3 servers ? It would be good if you could share zk logs. 2) Hope you have checked that "myid" file is correct in each server - that each server has a distinct server id. 3) Do you have firewall/security and no issues overthere ?. Make sure 2888/3888 are all open. 4) Hope /etc/hosts entries on all the nodes are fine. 5) Have you configured sasl configs in Observer nodes? Rakesh On Wed, Sep 26, 2018 at 9:19 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Any help? > > On Tue, Sep 25, 2018 at 2:20 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> And observer never joining the cluster its keep saying "Cannot open >> channel to" in the logs. >> >> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> Rakesh, >>> >>> Thank you, i have 3 floower and 3 observers in two different DC's >>> followers came up fine with SASL but for some reasons observers are not >>> coming up with the following error but i dont see any network issues, i was >>> able to telnet to 2181 and 3888 ports. >>> >>> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >>> size: 1 >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >>> size: 1 >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >>> size: 1 >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - >>> Opening channel to server 1 >>> 2018-09-24 17:55:34,151 [myid:6] - WARN >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot >>> open channel to 1 at election address zk-server1/10.16.1.102:3888 >>> java.net.SocketTimeoutException: connect timed out >>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>> at >>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >>> at >>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >>> at >>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >>> at java.net.Socket.connect(Socket.java:589) >>> at >>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) >>> at >>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) >>> at >>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) >>> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) >>> >>> >>> server.1=zk-server1:2888:3888 >>> server.2=zk-server2:2888:3888 >>> server.3=zk-server3:2888:3888 >>> server.4=zk-server4:2888:3888:observer >>> server.5=zk-server5:2888:3888:observer >>> server.6=zk-server6:2888:3888:observer >>> peerType=observer >>> >>> What could be the reason? >>> >>> Ram >>> >>> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan < >>> rake...@apache.org> wrote: >>> Thanks Ram for the interest on this feature. Yes, user can enable SASL for Observer nodes as well. In general, QuorumLearner will send authentication packet to peer QuorumServer. Observer is a learner which follows the same quorum authentication protocol and auth logic will work fine. FYI, hope you are referring below links for configurations, https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ Please let us know if you are facing any issues. Thanks, Rakesh On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Hi, > > Do we need to configure any thing on observer nodes for SASL > authentication? > > tcpKeepAlive=true ( this is not for sasl but just asking ) > > quorum.auth.enableSasl=true > quorum.auth.learnerRequireSasl=true > quorum.auth.serverRequireSasl=true > > What will happen if i set these properties on observers nodes as well ? > > Thanks, > Ram >
Re: Observer properties for SASL authentication in 3.4.13 version
Any help? On Tue, Sep 25, 2018 at 2:20 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > And observer never joining the cluster its keep saying "Cannot open > channel to" in the logs. > > On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> Rakesh, >> >> Thank you, i have 3 floower and 3 observers in two different DC's >> followers came up fine with SASL but for some reasons observers are not >> coming up with the following error but i dont see any network issues, i was >> able to telnet to 2181 and 3888 ports. >> >> >> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >> size: 1 >> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >> size: 1 >> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >> size: 1 >> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - Opening >> channel to server 1 >> 2018-09-24 17:55:34,151 [myid:6] - WARN >> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot >> open channel to 1 at election address zk-server1/10.16.1.102:3888 >> java.net.SocketTimeoutException: connect timed out >> at java.net.PlainSocketImpl.socketConnect(Native Method) >> at >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >> at >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >> at >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >> at java.net.Socket.connect(Socket.java:589) >> at >> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) >> at >> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) >> at >> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) >> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) >> >> >> server.1=zk-server1:2888:3888 >> server.2=zk-server2:2888:3888 >> server.3=zk-server3:2888:3888 >> server.4=zk-server4:2888:3888:observer >> server.5=zk-server5:2888:3888:observer >> server.6=zk-server6:2888:3888:observer >> peerType=observer >> >> What could be the reason? >> >> Ram >> >> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan >> wrote: >> >>> Thanks Ram for the interest on this feature. >>> >>> Yes, user can enable SASL for Observer nodes as well. In general, >>> QuorumLearner will send authentication packet to peer QuorumServer. >>> Observer is a learner which follows the same quorum authentication protocol >>> and auth logic will work fine. >>> >>> FYI, hope you are referring below links for configurations, >>> >>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication >>> >>> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ >>> >>> Please let us know if you are facing any issues. >>> >>> Thanks, >>> Rakesh >>> >>> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < >>> rammohanga...@gmail.com> wrote: >>> Hi, Do we need to configure any thing on observer nodes for SASL authentication? tcpKeepAlive=true ( this is not for sasl but just asking ) quorum.auth.enableSasl=true quorum.auth.learnerRequireSasl=true quorum.auth.serverRequireSasl=true What will happen if i set these properties on observers nodes as well ? Thanks, Ram >>>
Re: Observer properties for SASL authentication in 3.4.13 version
And observer never joining the cluster its keep saying "Cannot open channel to" in the logs. On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Rakesh, > > Thank you, i have 3 floower and 3 observers in two different DC's > followers came up fine with SASL but for some reasons observers are not > coming up with the following error but i dont see any network issues, i was > able to telnet to 2181 and 3888 ports. > > > 2018-09-24 17:55:34,145 [myid:6] - DEBUG > [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue > size: 1 > 2018-09-24 17:55:34,145 [myid:6] - DEBUG > [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue > size: 1 > 2018-09-24 17:55:34,145 [myid:6] - DEBUG > [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue > size: 1 > 2018-09-24 17:55:34,145 [myid:6] - DEBUG > [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - Opening > channel to server 1 > 2018-09-24 17:55:34,151 [myid:6] - WARN > [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot > open channel to 1 at election address zk-server1/10.16.1.102:3888 > java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) > at > org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) > at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) > > > server.1=zk-server1:2888:3888 > server.2=zk-server2:2888:3888 > server.3=zk-server3:2888:3888 > server.4=zk-server4:2888:3888:observer > server.5=zk-server5:2888:3888:observer > server.6=zk-server6:2888:3888:observer > peerType=observer > > What could be the reason? > > Ram > > On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan > wrote: > >> Thanks Ram for the interest on this feature. >> >> Yes, user can enable SASL for Observer nodes as well. In general, >> QuorumLearner will send authentication packet to peer QuorumServer. >> Observer is a learner which follows the same quorum authentication protocol >> and auth logic will work fine. >> >> FYI, hope you are referring below links for configurations, >> >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication >> >> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ >> >> Please let us know if you are facing any issues. >> >> Thanks, >> Rakesh >> >> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> Hi, >>> >>> Do we need to configure any thing on observer nodes for SASL >>> authentication? >>> >>> tcpKeepAlive=true ( this is not for sasl but just asking ) >>> >>> quorum.auth.enableSasl=true >>> quorum.auth.learnerRequireSasl=true >>> quorum.auth.serverRequireSasl=true >>> >>> What will happen if i set these properties on observers nodes as well ? >>> >>> Thanks, >>> Ram >>> >>
Re: Observer properties for SASL authentication in 3.4.13 version
Rakesh, Thank you, i have 3 floower and 3 observers in two different DC's followers came up fine with SASL but for some reasons observers are not coming up with the following error but i dont see any network issues, i was able to telnet to 2181 and 3888 ports. 2018-09-24 17:55:34,145 [myid:6] - DEBUG [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue size: 1 2018-09-24 17:55:34,145 [myid:6] - DEBUG [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue size: 1 2018-09-24 17:55:34,145 [myid:6] - DEBUG [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue size: 1 2018-09-24 17:55:34,145 [myid:6] - DEBUG [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - Opening channel to server 1 2018-09-24 17:55:34,151 [myid:6] - WARN [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot open channel to 1 at election address zk-server1/10.16.1.102:3888 java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) server.1=zk-server1:2888:3888 server.2=zk-server2:2888:3888 server.3=zk-server3:2888:3888 server.4=zk-server4:2888:3888:observer server.5=zk-server5:2888:3888:observer server.6=zk-server6:2888:3888:observer peerType=observer What could be the reason? Ram On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan wrote: > Thanks Ram for the interest on this feature. > > Yes, user can enable SASL for Observer nodes as well. In general, > QuorumLearner will send authentication packet to peer QuorumServer. > Observer is a learner which follows the same quorum authentication protocol > and auth logic will work fine. > > FYI, hope you are referring below links for configurations, > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication > > https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ > > Please let us know if you are facing any issues. > > Thanks, > Rakesh > > On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> Hi, >> >> Do we need to configure any thing on observer nodes for SASL >> authentication? >> >> tcpKeepAlive=true ( this is not for sasl but just asking ) >> >> quorum.auth.enableSasl=true >> quorum.auth.learnerRequireSasl=true >> quorum.auth.serverRequireSasl=true >> >> What will happen if i set these properties on observers nodes as well ? >> >> Thanks, >> Ram >> >
Re: Observer properties for SASL authentication in 3.4.13 version
Thanks Ram for the interest on this feature. Yes, user can enable SASL for Observer nodes as well. In general, QuorumLearner will send authentication packet to peer QuorumServer. Observer is a learner which follows the same quorum authentication protocol and auth logic will work fine. FYI, hope you are referring below links for configurations, https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ Please let us know if you are facing any issues. Thanks, Rakesh On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Hi, > > Do we need to configure any thing on observer nodes for SASL > authentication? > > tcpKeepAlive=true ( this is not for sasl but just asking ) > > quorum.auth.enableSasl=true > quorum.auth.learnerRequireSasl=true > quorum.auth.serverRequireSasl=true > > What will happen if i set these properties on observers nodes as well ? > > Thanks, > Ram >
Re: Observer properties for SASL authentication in 3.4.13 version
Ok, thanks On Mon, Sep 24, 2018 at 11:29 AM Norbert Kalmar wrote: > Unfortunately I'm not entirely sure on this one, and I can't test it out > right now, but shouldn't be any different then a normal follower. So you > should configure SASL the same way. The only difference basically is that > they are non-voters. Everything else works the same. Clients connect and > can send read / write commands. So it would be a huge security hole if an > observer is not configured as well. > > Regards, > Norbert > > On Mon, Sep 24, 2018 at 10:59 AM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > > > Any thoughts? > > > > On Sun, Sep 23, 2018 at 8:00 PM rammohan ganapavarapu < > > rammohanga...@gmail.com> wrote: > > > > > Hi, > > > > > > Do we need to configure any thing on observer nodes for SASL > > > authentication? > > > > > > tcpKeepAlive=true ( this is not for sasl but just asking ) > > > > > > quorum.auth.enableSasl=true > > > quorum.auth.learnerRequireSasl=true > > > quorum.auth.serverRequireSasl=true > > > > > > What will happen if i set these properties on observers nodes as well ? > > > > > > Thanks, > > > Ram > > > > > >
Re: Observer properties for SASL authentication in 3.4.13 version
Unfortunately I'm not entirely sure on this one, and I can't test it out right now, but shouldn't be any different then a normal follower. So you should configure SASL the same way. The only difference basically is that they are non-voters. Everything else works the same. Clients connect and can send read / write commands. So it would be a huge security hole if an observer is not configured as well. Regards, Norbert On Mon, Sep 24, 2018 at 10:59 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Any thoughts? > > On Sun, Sep 23, 2018 at 8:00 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > > > Hi, > > > > Do we need to configure any thing on observer nodes for SASL > > authentication? > > > > tcpKeepAlive=true ( this is not for sasl but just asking ) > > > > quorum.auth.enableSasl=true > > quorum.auth.learnerRequireSasl=true > > quorum.auth.serverRequireSasl=true > > > > What will happen if i set these properties on observers nodes as well ? > > > > Thanks, > > Ram > > >
Re: Observer properties for SASL authentication in 3.4.13 version
Any thoughts? On Sun, Sep 23, 2018 at 8:00 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Hi, > > Do we need to configure any thing on observer nodes for SASL > authentication? > > tcpKeepAlive=true ( this is not for sasl but just asking ) > > quorum.auth.enableSasl=true > quorum.auth.learnerRequireSasl=true > quorum.auth.serverRequireSasl=true > > What will happen if i set these properties on observers nodes as well ? > > Thanks, > Ram >
