[jira] [Comment Edited] (HADOOP-15250) Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460190#comment-16460190
]
Ajay Kumar edited comment on HADOOP-15250 at 5/1/18 10:02 PM:
--
[~gss2002] Thanks for reporting the issue and initial patch.
[~ste...@apache.org] Thanks for review and commit.
was (Author: ajayydv):
[~ste...@apache.org] Thanks for review and commit.
> Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
> --
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
> Environment: Multihome cluster with split DNS and rDNS lookup of
> localhost returning non-routable IPAddr
>Reporter: Greg Senia
>Assignee: Ajay Kumar
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: HADOOP-15250.00.patch, HADOOP-15250.01.patch,
> HADOOP-15250.02.patch, HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
> | *{color:#FF}{color}* |*{color:#FF}}{color}*|
> | |}|
> | |}|
>
> So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
> correctly out the correct interfaces:
>
> diff --git
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
>
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> index e1be271..c5b4a42 100644
> ---
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys
[jira] [Comment Edited] (HADOOP-15250) Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459644#comment-16459644
]
Steve Loughran edited comment on HADOOP-15250 at 5/1/18 12:00 PM:
--
LGTM on the basis that if the option isn't set, then all is as before. If you
change it, and Kerberos breaks, well, time for a new JIRA & patch. Key point: I
don't see a regression, and we do at least get a debug statement early on.
* XML Config text shouldn't say ::0 as that will create the expectation that we
support -IPv4- IPv6
Other than, unless someone wants to add docs on how to use this (anyone?). I'm
happy
was (Author: ste...@apache.org):
LGTM on the basis that if the option isn't set, then all is as before. If you
change it, and Kerberos breaks, well, time for a new JIRA & patch. Key point: I
don't see a regression, and we do at least get a debug statement early on.
* XML Config text shouldn't say ::0 as that will create the expectation that we
support IPv4.
Other than, unless someone wants to add docs on how to use this (anyone?). I'm
happy
> Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
> --
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
> Environment: Multihome cluster with split DNS and rDNS lookup of
> localhost returning non-routable IPAddr
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.00.patch, HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localA
[jira] [Comment Edited] (HADOOP-15250) Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[ https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372896#comment-16372896 ] Axton Grams edited comment on HADOOP-15250 at 2/22/18 3:11 PM: --- I work with Greg on the same clusters. To add some color to the DNS/split view configuration: * DNS is configured with 2 views: ** Internal: Used by cluster machines to resolve Hadoop nodes to cluster network segment IP ** External: Used by non-cluster machines to resolve Hadoop nodes to routable network segment IP * All nodes with a presence on the cluster network resolve machines to the cluster (non-routable) IP address * All nodes without a presence on the cluster network resolve machines to the routable IP address We implemented this pattern for the following reasons: * We can allow unfettered access (iptables/firewalld) between cluster nodes * We use jumbo frames on the cluster network to ease network load You have to understand that the interface the service binds to is conditional depending on the origin of the traffic, not how the server knows itself according to DNS or Kerberos. Different nodes know the server by the same name with different IP addresses, depending on whether they have a presence on the cluster network segment. All Hadoop nodes know themselves by the cluster IP address, which is non-routable. This design is compatible with the Linux network stack, DNS view practices, multi-homing practices, and all other related technology domains, just not Hadoop. We operate with the following assumptions: * The network stack provided by the OS knows how to properly route traffic * The information in DNS is properly managed and accurate * The hostname matches the Kerberos principal name, but the IP answer is different different for different clients was (Author: agrams): I work with Greg on the same clusters. To add some color to the DNS/split view configuration: * DNS is configured with 2 views: ** Internal: Used by cluster machines to resolve Hadoop nodes to cluster network segment IP ** External: Used by non-cluster machines to resolve Hadoop nodes to routable network segment IP * All nodes with a presence on the cluster network resolve machines to the cluster (non-routable) IP address * All nodes without a presence on the cluster network resolve machines to the routable IP address We implemented this pattern for the following reasons: * We can allow unfettered access (iptables/firewalld) between cluster nodes * We use jumbo frames on the cluster network to ease network load You have to understand that the interface the service binds to is conditional depending on the origin of the traffic, not how the server knows itself according to DNS or Kerberos. Different nodes know the server by the same name with different IP addresses, depending on whether they have a presence on the cluster network segment. All Hadoop nodes know themselves by the cluster IP address, which is non-routable. This design is compatible with the Linux network stack, DNS view practices, multi-homing practices, and all other related technology domains, just not Hadoop. > Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong > -- > > Key: HADOOP-15250 > URL: https://issues.apache.org/jira/browse/HADOOP-15250 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, net >Affects Versions: 2.7.3, 2.9.0, 3.0.0 > Environment: Multihome cluster with split DNS and rDNS lookup of > localhost returning non-routable IPAddr >Reporter: Greg Senia >Priority: Critical > Attachments: HADOOP-15250.patch > > > We run our Hadoop clusters with two networks attached to each node. These > network are as follows a server network that is firewalled with firewalld > allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and > the HTTP YARN RM/ATS and MR History Server. The second network is the cluster > network on the second network interface this uses Jumbo frames and is open no > restrictions and allows all cluster traffic to flow between nodes. > > To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the > traffic is originating from nodes with cluster networks we return the > internal DNS record for the nodes. This all works fine with all the > multi-homing features added to Hadoop 2.x > Some logic around views: > a. The internal view is used by cluster machines when performing lookups. So > hosts on the cluster network should get answers from the internal view in DNS > b. The external view is used by non-local-cluster machines when performing > lookups. So hosts not on the cluster network should get answers from the > e
[jira] [Comment Edited] (HADOOP-15250) Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[ https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372818#comment-16372818 ] Steve Loughran edited comment on HADOOP-15250 at 2/22/18 2:16 PM: -- [~ste...@apache.org] I think by binding to what is resolved by DNS you inherently break the ability to do Multi-homing with a routable and non-routable networks where you have split view DNS as in our case. We actually ran this design two years ago up the support chain of our Hadoop Vendor and was passed as being just fine. So the thought with splitview DNS is as follows: the server interface is resolved to ha21d52mn.unit.hdp.example.com when outside the cluster when inside the cluster ha21d52mn.unit.hdp.newyorklife.com is resolved to the cluster networks interface this was why we went with the DNS split views to support multi-homing correctly. If this functionality is NOT supported than Hadoop Project should remove the multi-homing features as things going to remote clusters will not work as shown by my trace/logs above. As Unit/Dev Cluster Network is not routable to our Tech/Test Cluster Network. So traffic would have to go out the server interfaces to get to another cluster hence why the splitview DNS is valid and a fix along these lines should at least allow a flag like the server components do to bind to 0.0.0.0. I do see HADOOP-7215 is what introduced this but I don't think multi-homing was thought about when this code was implemented and this code is really is only checking for null's. So if this is not the right place than I think a solution needs to be determined to determine how to allow a Client to bind without using a specific IP/hostname as that is not a valid method with a multi-homed network. Also hadoop traffic destined for other hosts in the cluster would go via the cluster network and allow from use of jumbo frames as its not routable. So when my co-workers and I decided to test this scenario out we modifed the code to allow traffic to bind to a valid local address depending on where the traffic needs to be destined this patch is working in our tech/test cluster. As the java.net.Socket doc shows you can run bind and it will bind to a valid local address. I mean attempting to bind outside of the OS opens up risks like we are hitting with distcp attempting to bind to an invalid source address that is non-routable [https://docs.oracle.com/javase/8/docs/api/java/net/Socket.html#bind-java.net.SocketAddress-] was (Author: gss2002): [~ste...@apache.org] I think by binding to what is resolved by DNS you inherently break the ability to do Multi-homing with a routable and non-routable networks where you have split view DNS as in our case. We actually ran this design two years ago up the support chain of our Hadoop Vendor and was passed as being just fine. So the thought with splitview DNS is as follows: the server interface is resolved to ha21d52mn.unit.hdp.example.com when outside the cluster when inside the cluster ha21d52mn.unit.hdp.newyorklife.com is resolved to the cluster networks interface this was why we went with the DNS split views to support multi-homing correctly. If this functionality is NOT supported than Hadoop Project should remove the multi-homing features as things going to remote clusters will not work as shown by my trace/logs above. As Unit/Dev Cluster Network is not routable to our Tech/Test Cluster Network. So traffic would have to go out the server interfaces to get to another cluster hence why the splitview DNS is valid and a fix along these lines should at least allow a flag like the server components do to bind to 0.0.0.0. I do see HADOOP-7215 is what introduced this but I don't think multi-homing was thought about when this code was implemented and this code is really is only checking for null's. So if this is not the right place than I think a solution needs to be determined to determine how to allow a Client to bind without using a specific IP/hostname as that is not a valid method with a multi-homed network. Also hadoop traffic destined for other hosts in the cluster would go via the cluster network and allow from use of jumbo frames as its not routable. So when my co-workers and I decided to test this scenario out we modifed the code to allow traffic to bind to a valid local address depending on where the traffic needs to be destined this patch is working in our tech/test cluster. As the java.net.Socket doc shows you can run bind and it will bind to a valid local address. I mean attempting to bind outside of the OS opens up risks like we are hitting with distcp attempting to bind to an invalid source address that is non-routable [https://docs.oracle.com/javase/8/docs/api/java/net/Socket.html#bind-java.net.SocketAddress-] public void bind( [SocketAddress|https://docs.oracle.com/javase/8/docs/api/
