[jira] [Commented] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372818#comment-16372818
]
Greg Senia commented on HADOOP-15250:
-
[~ste...@apache.org] I think by binding to what is resolved by DNS you
inherently break the ability to do Multi-homing with a routable and
non-routable networks where you have split view DNS as in our case. We actually
ran this design two years ago up the support chain of our Hadoop Vendor and was
passed as being just fine. So the thought with splitview DNS is as follows: the
server interface is resolved to ha21d52mn.unit.hdp.example.com when outside the
cluster when inside the cluster ha21d52mn.unit.hdp.newyorklife.com is resolved
to the cluster networks interface this was why we went with the DNS split views
to support multi-homing correctly. If this functionality is NOT supported than
Hadoop Project should remove the multi-homing features as things going to
remote clusters will not work as shown by my trace/logs above. As Unit/Dev
Cluster Network is not routable to our Tech/Test Cluster Network. So traffic
would have to go out the server interfaces to get to another cluster hence why
the splitview DNS is valid and a fix along these lines should at least allow a
flag like the server components do to bind to 0.0.0.0. I do see HADOOP-7215 is
what introduced this but I don't think multi-homing was thought about when this
code was implemented and this code is really is only checking for null's. So
if this is not the right place than I think a solution needs to be determined
to determine how to allow a Client to bind without using a specific IP/hostname
as that is not a valid method with a multi-homed network. Also hadoop traffic
destined for other hosts in the cluster would go via the cluster network and
allow from use of jumbo frames as its not routable.
So when my co-workers and I decided to test this scenario out we modifed the
code to allow traffic to bind to a valid local address depending on where the
traffic needs to be destined this patch is working in our tech/test cluster. As
the java.net.Socket doc shows you can run bind and it will bind to a valid
local address. I mean attempting to bind outside of the OS opens up risks like
we are hitting with distcp attempting to bind to an invalid source address that
is non-routable
[https://docs.oracle.com/javase/8/docs/api/java/net/Socket.html#bind-java.net.SocketAddress-]
public void bind(
[SocketAddress|https://docs.oracle.com/javase/8/docs/api/java/net/SocketAddress.html]
bindpoint) throws
[IOException|https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html]
Binds the socket to a local address.If the address is {{null}}, then the system
will pick up an ephemeral port and a valid local address to bind the socket.
|/**|
| | * Checks if {@code host} is a local host name and return {@link
InetAddress}|
| | * corresponding to that address.|
| | * |
| | * @param host the specified host|
| | * @return a valid local {@link InetAddress} or null|
| | * @throws SocketException if an I/O error occurs|
| | */|
| |public static InetAddress getLocalInetAddress(String host)|
| |throws SocketException {|
| |if (host == null) {|
| |return null;|
| |}|
| |InetAddress addr = null;|
| |try {|
| |addr = SecurityUtil.getByName(host);|
| |if (NetworkInterface.getByInetAddress(addr) == null) {|
| |addr = null; // Not a local address|
| |}|
| |} catch (UnknownHostException ignore) \{ }|
| |return addr;|
| |}|
> MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
>
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get
[jira] [Commented] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372693#comment-16372693
]
Steve Loughran commented on HADOOP-15250:
-
I'm not going to review this (sorry), as I don't have enough knowledge of the
deep IPC internals to be safe to. Sorry
But I do know that we try to bind up on the hostname of the kerberos principal,
so that your service is who it claims to be, the caller can avoid MITM
problems. This kind of opens things up again, doesn't it?
> MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
>
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
> | *{color:#FF}{color}* |*{color:#FF}}{color}*|
> | |}|
> | |}|
>
> So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
> correctly out the correct interfaces:
>
> diff --git
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
>
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> index e1be271..c5b4a42 100644
> ---
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> +++
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> @@ -305,6 +305,9 @@
> public static final String
[jira] [Commented] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372201#comment-16372201
]
Greg Senia commented on HADOOP-15250:
-
To test this patch as a unit test you would need to have a multi-homed network
with one routable interface that only external traffic travels over and one
cluster network interface that is not-routable
> MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
>
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
> | *{color:#FF}{color}* |*{color:#FF}}{color}*|
> | |}|
> | |}|
>
> So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
> correctly out the correct interfaces:
>
> diff --git
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
>
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> index e1be271..c5b4a42 100644
> ---
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> +++
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> @@ -305,6 +305,9 @@
> public static final String IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY
> = "ipc.client.fallback-to-simple-auth-allowed";
> public static final boolean
>
[jira] [Commented] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372188#comment-16372188
]
genericqa commented on HADOOP-15250:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 3 new + 176 unchanged - 0 fixed = 179 total (was 176) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
36s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15250 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911432/HADOOP-15250.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c523ea371d06 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2bc3351 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14181/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14181/testReport/ |
| Max. process+thread count | 1572 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372050#comment-16372050
]
Greg Senia commented on HADOOP-15250:
-
Interface Information. The Interface *{color:#33}eno33559296{color}* is the
cluster interface which is non-routable. Interface *eno16780032* ** is the
server network publically accessible interface. No Hadoop Traffic flows over
this interface unless its traffic originating/terminating outside of the Hadoop
Cluster. I've also included the routing table below and a network trace showing
the wrong IP being used due to the fact that Client.java binds outbound calls
to the hostname it finds in DNS or /etc/hosts.
[root@ha21d52mn yarn]# ifconfig -a
{color:#33}*eno16780032: flags=4163 mtu
1500*{color}
{color:#33} *inet 10.69.81.1 netmask 255.255.240.0 broadcast
10.69.95.255*{color}
inet6 fe80::250:56ff:fe82:4934 prefixlen 64 scopeid 0x20
ether 00:50:56:82:49:34 txqueuelen 1000 (Ethernet)
RX packets 84514982 bytes 25768335637 (23.9 GiB)
RX errors 0 dropped 6172 overruns 0 frame 0
TX packets 83332181 bytes 18600190794 (17.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
{color:#33}*eno33559296: flags=4163 mtu
9000*{color}
{color:#33} *inet 10.70.49.1 netmask 255.255.240.0 broadcast
10.70.63.255*{color}
inet6 fe80::250:56ff:fe82:379c prefixlen 64 scopeid 0x20
ether 00:50:56:82:37:9c txqueuelen 1000 (Ethernet)
RX packets 1649741562 bytes 868670646085 (809.0 GiB)
RX errors 0 dropped 5052 overruns 0 frame 0
TX packets 1248707764 bytes 1782972383010 (1.6 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ha21d52mn yarn]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
*default gateway 0.0.0.0 UG 0 0 0
eno16780032*
*10.69.80.0 0.0.0.0 255.255.240.0 U 0 0 0
eno16780032*
*10.70.48.0 0.0.0.0 255.255.240.0 U 0 0 0
eno33559296*
link-local 0.0.0.0 255.255.0.0 U 1002 0 0
eno16780032
link-local 0.0.0.0 255.255.0.0 U 1003 0 0
eno33559296
Here is an Example of the attempt to send traffic using the wrong interface
before the patch:
[root@ha21d52mn yarn]# tcpdump -s0 -i eno16780032 -nn host ha21t51nn.tech.hdp
or host ha21t52nn.tech.hdp and port 8020
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno16780032, link-type EN10MB (Ethernet), capture size 65535 bytes
16:27:35.656967 IP 10.70.49.1.34065 > 10.69.49.7.8020: Flags [S], seq 5653335,
win 29200, options [mss 1460,sackOK,TS val 3923129189 ecr 0,nop,wscale 7],
length 0
16:27:36.659542 IP 10.70.49.1.34065 > 10.69.49.7.8020: Flags [S], seq 5653335,
win 29200, options [mss 1460,sackOK,TS val 3923130192 ecr 0,nop,wscale 7],
length 0
16:27:38.663551 IP *10.70.49.1*.34065 > 10.69.49.7.8020: Flags [S], seq
5653335, win 29200, options [mss 1460,sackOK,TS val 3923132196 ecr 0,nop,wscale
7], length 0
16:27:42.675539 IP 10.70.49.1.34065 > 10.69.49.7.8020: Flags [S], seq 5653335,
win 29200, options [mss 1460,sackOK,TS val 3923136208 ecr 0,nop,wscale 7],
length 0
^C
2018-02-21 16:23:55,075 INFO retry.RetryInvocationHandler
(RetryInvocationHandler.java:log(267)) - Exception while invoking
ClientNamenodeProtocolTranslatorPB.renewDelegationToken over
ha21t52nn.tech.hdp.example.com/10.69.49.7:8020 after 1 failover attempts.
Trying to failover after sleeping for 1290ms.
org.apache.hadoop.net.ConnectTimeoutException: Call From
ha21d52mn.unit.hdp.example.com/10.70.49.1 to
ha21t52nn.tech.hdp.example.com:8020 failed on socket timeout exception:
org.apache.hadoop.net.ConnectTimeoutException: 2 millis timeout while
waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending local=/10.70.49.1:35231
remote=ha21t52nn.tech.hdp.example.com/10.69.49.7:8020]; For more details see:
http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.GeneratedConstructorAccessor205.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at
