-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63872/
-----------------------------------------------------------
Review request for Ambari, Sid Wagle and Vitalyi Brodetskyi.
Bugs: AMBARI-22460
https://issues.apache.org/jira/browse/AMBARI-22460
Repository: ambari
Description
-------
Initializing privileged NFS client socket...
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: java.net.ConnectException: Call From
ctr-e134-1499953498516-310520-01-000002.hwx.site/172.27.26.203 to
ctr-e134-1499953498516-310520-01-000007.hwx.site:8020 failed on connection
exception: java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1498)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:823)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2165)
at
org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1442)
at
org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1438)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1454)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:770)
at
org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd.addExports(RpcProgramMountd.java:107)
at
org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd.<init>(RpcProgramMountd.java:94)
at org.apache.hadoop.hdfs.nfs.mount.Mountd.<init>(Mountd.java:37)
at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.<init>(Nfs3.java:47)
at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:67)
at
org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter.start(PrivilegedNfsGatewayStarter.java:71)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
NFSGateway tries to connect to Namenode during start up. But in this case
NFSGateway starts before Namenode. We have no dependency in
role_command_order.json to regulate this. (Ps. deploy is non-blueprint)
As to how this worked before.
Most likely the ordering for was good because of how ordering algorithm worked.
However maybe QE tests were adding NFS Gateway component only after Initial
Deploy is done. Or maybe nfsgateway had some retry connect to NN code which is
broken now.
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.0.6/role_command_order.json
78a31f1797
Diff: https://reviews.apache.org/r/63872/diff/1/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
