Hi folks, I've just upgraded our Samza app to v1.5.1. Everything is building and running correctly, and the command to load it into our YARN cluster is initially succeeding. However, as the application is starting in YARN, this message is produced repeatedly until it hits a retry threshold:
"logger_name": "org.apache.hadoop.ipc.Client","message":"Retrying connect to server: localhost/127.0.0.1:8030. Already tried # time(s)" As I said, this is coming from an application that is running on the YARN server itself, so we seem to be able to use the local yarn-site.xml accurately. When it starts on yarn, though, it seems to be resolving the resource manager to localhost. The final exception information is at the end of this email. Any ideas? Cheers, Malcolm McFarland Cavulus Failed to connect to server: localhost/127.0.0.1:8030: retries get failed due to exceeded maximum allowed retries number: 10 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788) at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550) at org.apache.hadoop.ipc.Client.call(Client.java:1381) at org.apache.hadoop.ipc.Client.call(Client.java:1345) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy48.registerApplicationMaster(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:107) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) at com.sun.proxy.$Proxy49.registerApplicationMaster(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:223) at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:214) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:139) at org.apache.samza.job.yarn.SamzaYarnAppMasterLifecycle.onInit(SamzaYarnAppMasterLifecycle.scala:42) at org.apache.samza.job.yarn.YarnClusterResourceManager.start(YarnClusterResourceManager.java:218) at org.apache.samza.clustermanager.ContainerProcessManager.start(ContainerProcessManager.java:230) at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.run(ClusterBasedJobCoordinator.java:289) at org.apache.samza.clustermanager.JobCoordinatorLaunchUtil.run(JobCoordinatorLaunchUtil.java:83) at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.runClusterBasedJobCoordinator(ClusterBasedJobCoordinator.java:529) at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.main(ClusterBasedJobCoordinator.java:473)
