[jira] [Comment Edited] (ZOOKEEPER-2982) Re-try DNS hostname -> IP resolution
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393254#comment-16393254 ] Eron Wright edited comment on ZOOKEEPER-2982 at 3/9/18 5:40 PM: - [~andorm] are you driving this issue now? Would you please assign the bug appropriately? I'm keen to see the patch make it into 3.5.4. was (Author: eronwright): [~andorm] are you driving this issue now? Would you please assign the bug appropriate? I'm keen to see the patch make it into 3.5.4. > Re-try DNS hostname -> IP resolution > > > Key: ZOOKEEPER-2982 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2982 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.0, 3.5.1, 3.5.3 >Reporter: Eron Wright >Priority: Blocker > Fix For: 3.5.4, 3.6.0 > > Attachments: 3.5.3-beta.zip, fixed.log > > > ZOOKEEPER-1506 fixed a DNS resolution issue in 3.4. Some portions of the fix > haven't yet been ported to 3.5. > To recap the outstanding problem in 3.5, if a given ZK server is started > before all peer addresses are resolvable, that server may cache a negative > lookup result and forever fail to resolve the address.For example, > deploying ZK 3.5 to Kubernetes using a StatefulSet plus a Service (headless) > may fail because the DNS records are created lazily. > {code} > 2018-02-18 09:11:22,583 [myid:0] - WARN > [QuorumPeer[myid=0](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@95] > - Exception when following the leader > java.net.UnknownHostException: zk-2.zk.default.svc.cluster.local > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at > org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227) > at > org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256) > at > org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:76) > at > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1133) > {code} > In the above example, the address `zk-2.zk.default.svc.cluster.local` was not > resolvable when the server started, but became resolvable shortly thereafter. >The server should eventually succeed but doesn't. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ZOOKEEPER-2982) Re-try DNS hostname -> IP resolution
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375213#comment-16375213 ] Abraham Fine edited comment on ZOOKEEPER-2982 at 2/24/18 1:42 AM: -- [~fpj] I believe your diagnosis to be correct and I agree that [~eronwright]'s fix would solve the problem in the case that DNS eventually is fixed. My concern with the current solution is that it could cause us to jump back and forth between leader election and the quorum when the DNS stays in a bad state. For example, imagine a 3 node cluster {z1, z2, z3}. z3 is always offline and z2 has no entry in dns. z2 will connect to z1 and win the leader election. When it comes time to form the quorum z1 will be unable to follow z2 as it wont be able to resolve its address. Just spitballing here, but what if we had z1 connect to the {{remoteSocketAddress}} of the socket created from the connection it received in {{QuorumCnxManager}}? I understand there are some security concerns here and I'm not sure how much we care about that since they would be cured by Kerberos or TLS (one day). We could also do a reverse dns lookup and reject the connection if the reverse lookup does not align with our expected hostname. What do you guys think? was (Author: abrahamfine): [~fpj] I believe your diagnosis to be correct and I agree that [~eronwright]'s fix would solve the problem in the case that DNS eventually is fixed. My concern with the current solution is that it could cause us to jump back and forth between leader election and the quorum when the DNS stays in a bad state. For example, imagine a 3 node cluster {z1, z2, z3}. z3 is always offline and z2 has no entry in dns. z2 will connect to z1 and win the leader election. When it comes time to form the quorum z1 will be unable to follow z2 as it wont be able to resolve its address. Just spitballing here, but what if we had z1 connect to the {{remoteSocketAddress}} of the socket created from the connection it received in {{QuorumCnxManager}}? I understand there are some security concerns here and I'm not sure how much we care about that since they would be stifled by Kerberos. We could also do a reverse dns lookup and reject the connection if the reverse lookup does not align with our expected hostname. What do you guys think? > Re-try DNS hostname -> IP resolution > > > Key: ZOOKEEPER-2982 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2982 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.0, 3.5.1, 3.5.3 >Reporter: Eron Wright >Priority: Blocker > Fix For: 3.5.4, 3.6.0 > > Attachments: 3.5.3-beta.zip, fixed.log > > > ZOOKEEPER-1506 fixed a DNS resolution issue in 3.4. Some portions of the fix > haven't yet been ported to 3.5. > To recap the outstanding problem in 3.5, if a given ZK server is started > before all peer addresses are resolvable, that server may cache a negative > lookup result and forever fail to resolve the address.For example, > deploying ZK 3.5 to Kubernetes using a StatefulSet plus a Service (headless) > may fail because the DNS records are created lazily. > {code} > 2018-02-18 09:11:22,583 [myid:0] - WARN > [QuorumPeer[myid=0](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@95] > - Exception when following the leader > java.net.UnknownHostException: zk-2.zk.default.svc.cluster.local > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at > org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227) > at > org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256) > at > org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:76) > at > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1133) > {code} > In the above example, the address `zk-2.zk.default.svc.cluster.local` was not > resolvable when the server started, but became resolvable shortly thereafter. >The server should eventually succeed but doesn't. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ZOOKEEPER-2982) Re-try DNS hostname -> IP resolution
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372542#comment-16372542 ] Flavio Junqueira edited comment on ZOOKEEPER-2982 at 2/22/18 8:33 AM: -- I have tried your recipe for reproducing as well [~andorm] by changing {{/etc/hosts}} and got the same issue. The problem is that the leader fails to bind to the port, which actually makes me wonder whether we need to do anything about the leader with respect to this issue: {noformat} java.net.SocketException: Unresolved address at java.net.ServerSocket.bind(ServerSocket.java:368) at java.net.ServerSocket.bind(ServerSocket.java:329) at org.apache.zookeeper.server.quorum.Leader.(Leader.java:240) at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:1023) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1226) {noformat} Your suggestion of the alternative change is sensible, but I'd say that for consistency, it is better that we simply do the same that we have in 3.4, which is to make the change in {{findLeader}}. One thing that I believe we haven't been able to do is to have a test case to report it. It would be good to have it, but I'm not sure what would be a good way. was (Author: fpj): I have tried your recipe for reproducing as well [~andorm] by changing {{/etc/hosts}} and got the same issue. The problem is that the leader fails to bind to the port, which actually makes me wonder whether we need to do anything about the leader with respect to this issue: ``` java.net.SocketException: Unresolved address at java.net.ServerSocket.bind(ServerSocket.java:368) at java.net.ServerSocket.bind(ServerSocket.java:329) at org.apache.zookeeper.server.quorum.Leader.(Leader.java:240) at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:1023) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1226) ``` Your suggestion of the alternative change is sensible, but I'd say that for consistency, it is better that we simply do the same that we have in 3.4, which is to make the change in {{findLeader}}. One thing that I believe we haven't been able to do is to have a test case to report it. It would be good to have it, but I'm not sure what would be a good way. > Re-try DNS hostname -> IP resolution > > > Key: ZOOKEEPER-2982 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2982 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.0, 3.5.1, 3.5.3 >Reporter: Eron Wright >Priority: Blocker > Fix For: 3.5.4, 3.6.0 > > Attachments: 3.5.3-beta.zip, fixed.log > > > ZOOKEEPER-1506 fixed a DNS resolution issue in 3.4. Some portions of the fix > haven't yet been ported to 3.5. > To recap the outstanding problem in 3.5, if a given ZK server is started > before all peer addresses are resolvable, that server may cache a negative > lookup result and forever fail to resolve the address.For example, > deploying ZK 3.5 to Kubernetes using a StatefulSet plus a Service (headless) > may fail because the DNS records are created lazily. > {code} > 2018-02-18 09:11:22,583 [myid:0] - WARN > [QuorumPeer[myid=0](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@95] > - Exception when following the leader > java.net.UnknownHostException: zk-2.zk.default.svc.cluster.local > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at > org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227) > at > org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256) > at > org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:76) > at > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1133) > {code} > In the above example, the address `zk-2.zk.default.svc.cluster.local` was not > resolvable when the server started, but became resolvable shortly thereafter. >The server should eventually succeed but doesn't. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ZOOKEEPER-2982) Re-try DNS hostname -> IP resolution
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371980#comment-16371980 ] Andor Molnar edited comment on ZOOKEEPER-2982 at 2/21/18 8:42 PM: -- Looks like that sock.connect() in connectToLeader() requires the address to be resolved already. If QuorumCnxnManager() fails to do that, connectToLeader() should be able to detect & fix it by doing resolution explicitly when addr.isUnresolved() == true. Not sure if it's any better than doing recreateSocketAddress() in findLeader(), but it may be another option to consider. was (Author: andorm): Looks like that sock.connect() in connectToLeader() requires the address to be resolved already. If QuorumCnxnManager() fails to do that, connectToLeader() might be able to detect & fix it by doing resolution explicitly when addr.isUnresolved() == true. Not sure if it's any better than doing recreateSocketAddress() in findLeader(), but it may be another option to consider. > Re-try DNS hostname -> IP resolution > > > Key: ZOOKEEPER-2982 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2982 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.0, 3.5.1, 3.5.3 >Reporter: Eron Wright >Priority: Blocker > Fix For: 3.5.4, 3.6.0 > > Attachments: 3.5.3-beta.zip, fixed.log > > > ZOOKEEPER-1506 fixed a DNS resolution issue in 3.4. Some portions of the fix > haven't yet been ported to 3.5. > To recap the outstanding problem in 3.5, if a given ZK server is started > before all peer addresses are resolvable, that server may cache a negative > lookup result and forever fail to resolve the address.For example, > deploying ZK 3.5 to Kubernetes using a StatefulSet plus a Service (headless) > may fail because the DNS records are created lazily. > {code} > 2018-02-18 09:11:22,583 [myid:0] - WARN > [QuorumPeer[myid=0](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@95] > - Exception when following the leader > java.net.UnknownHostException: zk-2.zk.default.svc.cluster.local > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at > org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227) > at > org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256) > at > org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:76) > at > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1133) > {code} > In the above example, the address `zk-2.zk.default.svc.cluster.local` was not > resolvable when the server started, but became resolvable shortly thereafter. >The server should eventually succeed but doesn't. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ZOOKEEPER-2982) Re-try DNS hostname -> IP resolution
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370631#comment-16370631 ] Abraham Fine edited comment on ZOOKEEPER-2982 at 2/20/18 10:28 PM: --- I'm wondering if [~rthille] can chime in on this. It looks like the change this JIRA is talking about is referenced by https://issues.apache.org/jira/browse/ZOOKEEPER-1506?focusedCommentId=14711955&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14711955 Is there a reason why this change was left out of branch-3.5 (and master)? My guess is that in master and branch-3.5 we always call `recreateSocketAddresses` in `connectOne` which should be called during leader election of communication to another quorum member stops. Again, it would be great to have [~rthille] confirm/tell me how wrong I am. was (Author: abrahamfine): I'm wondering if [~rthille] can chime in on this. It looks like the change this JIRA is talking about is referenced by https://issues.apache.org/jira/browse/ZOOKEEPER-1506?focusedCommentId=14711955&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14711955 Is there a reason why this change was left out of branch-3.5 (and master)? > Re-try DNS hostname -> IP resolution > > > Key: ZOOKEEPER-2982 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2982 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.0, 3.5.1, 3.5.3 >Reporter: Eron Wright >Priority: Blocker > Fix For: 3.5.4, 3.6.0 > > > ZOOKEEPER-1506 fixed a DNS resolution issue in 3.4. Some portions of the fix > haven't yet been ported to 3.5. > To recap the outstanding problem in 3.5, if a given ZK server is started > before all peer addresses are resolvable, that server may cache a negative > lookup result and forever fail to resolve the address.For example, > deploying ZK 3.5 to Kubernetes using a StatefulSet plus a Service (headless) > may fail because the DNS records are created lazily. > {code} > 2018-02-18 09:11:22,583 [myid:0] - WARN > [QuorumPeer[myid=0](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@95] > - Exception when following the leader > java.net.UnknownHostException: zk-2.zk.default.svc.cluster.local > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at > org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227) > at > org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256) > at > org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:76) > at > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1133) > {code} > In the above example, the address `zk-2.zk.default.svc.cluster.local` was not > resolvable when the server started, but became resolvable shortly thereafter. >The server should eventually succeed but doesn't. -- This message was sent by Atlassian JIRA (v7.6.3#76005)