Jagadish created SAMZA-1824:
-------------------------------
Summary: Samza AM does not handle some failures during container
launch
Key: SAMZA-1824
URL: https://issues.apache.org/jira/browse/SAMZA-1824
Project: Samza
Issue Type: Bug
Reporter: Jagadish
I noticed this behavior in the AM today where it fails to allocate a new
container on failure to start. The AM seems to get a callback for a container
failure during container startup and is never rescheduled again by the AM. The
logs seem to suggest that the request was made by the AM to start container
`container_e20_1528615592911_1987_02_000062` as seen below:
{code}
2018-08-06 15:39:46.369 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Received launch request for 12 on hostname lca1-app0596.stg.linkedin.com
2018-08-06 15:39:46.974 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Got available container ID (12) for container: Container: [ContainerId:
container_e20_1528615592911_1987_02_000062, NodeId:
lca1-app0596.stg.linkedin.com:1158, NodeHttpAddress:
lca1-app0596.stg.linkedin.com:8042, Resource: <memory:4096, vCores:1>,
Priority: 1, Token: Token { kind: ContainerToken, service: 10.251.166.210:1158
}, ]
2018-08-06 15:39:46.974 [Container Allocator Thread] YarnClusterResourceManager
[INFO] In runContainer in util: fwkPath= ;cmdPath=./__package/;jobLib=
2018-08-06 15:39:46.974 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Container ID 12 using command ./__package//bin/run-container.sh
2018-08-06 15:39:46.975 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Container ID 12 using environment variables:
SAMZA_CONTAINER_ID=12
EXECUTION_ENV_CONTAINER_ID=container_e20_1528615592911_1987_02_000062
SAMZA_COORDINATOR_URL=http://lca1-app1576.stg.linkedin.com:44222/
JAVA_OPTS=-Xmx3072m -Dcom.linkedin.app.name=ad-web-analytics-event-aggregator
-Dcom.linkedin.app.env=ei-lca1
{code}
But due to the issue where the localization phase fails (currently under
investigation) the container fails to start, throwing the following exception
to the `onStartContainerError` callback in `YarnClusterResourceManager`.
{code}
2018-08-06 15:46:15.257
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #6]
YarnClusterResourceManager [ERROR] Container:
container_e20_1528615592911_1987_02_000062 could not start.
java.net.ConnectException: Call From
lca1-app1576.stg.linkedin.com/10.251.174.104 to
lca1-app0596.stg.linkedin.com:1158 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor374.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
at org.apache.hadoop.ipc.Client.call(Client.java:1473)
at org.apache.hadoop.ipc.Client.call(Client.java:1400)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy34.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy35.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:201)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StartContainerTransition.transition(NMClientAsyncImpl.java:377)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StartContainerTransition.transition(NMClientAsyncImpl.java:363)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer.handle(NMClientAsyncImpl.java:498)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$ContainerEventProcessor.run(NMClientAsyncImpl.java:557)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:608)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:706)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:369)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
at org.apache.hadoop.ipc.Client.call(Client.java:1439)
... 22 more
2018-08-06 15:46:15.259
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #6]
YarnClusterResourceManager [INFO] Got an invalid notification for container:
container_e20_1528615592911_1987_02_000062
{code}
Looking at the code in `onStartContainerError`: it seems like we are picking a
container from a Map called ` containersPendingStartup` which doesn’t seem to
have valid containers (quick code search did not show when this map is
populated). This causes these failure callbacks to go unaddressed leaving the
job to have containers that are always pending and the only resolution is to
bounce the job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
