Hi all, I'm investigating the rolling upgrade process from Hadoop 2.6 to Hadoop 2.9.1. I'm trying to upgrade ResourceManager first and then try to upgrade NodeManager. When I submit a yarn job, RM fails with the following exception:
Application application_1549408943468_0001 failed 2 times due to Error launching appattempt_1549408943468_0001_000002. Got exception: java.io.IOException: Failed on local exception: java.io.IOException: java.io.EOFException; Host Details : local host is: "hadoopbenchaqjm01-sjc1/10.67.2.171"; destination host is: "hadoopbencha22-sjc1.prod.uber.internal":8041; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:805) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) at org.apache.hadoop.ipc.Client.call(Client.java:1439) at org.apache.hadoop.ipc.Client.call(Client.java:1349) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy87.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:128) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy88.startContainers(Unknown Source) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:122) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:307) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: java.io.EOFException at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1889) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813) at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:411) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1554) at org.apache.hadoop.ipc.Client.call(Client.java:1385) ... 20 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1798) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:365) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:615) at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:411) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:800) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:796) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1889) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795) ... 23 more and NM with 2019-02-06 00:29:20,214 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.67.2.171:54588:null (DIGEST-MD5: IO error acquiring password) with true cause: (null) I'm wondering if it's a known issue and anybody has an insight for it. Thanks, Aihua
