[
https://issues.apache.org/jira/browse/SAMZA-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadish updated SAMZA-1371:
----------------------------
Description:
I noticed this behavior in the AM today where it fails to allocate a new
container on failure to start. The AM seems to get a callback for a container
failure during container startup and is never rescheduled again by the AM. The
logs seem to suggest that the request was made by the AM to start container
`container_e20_1528615592911_1987_02_000062` as seen below:
{code}
2018-08-06 15:39:46.369 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Received launch request for 12 on hostname lca1-app0596.stg.linkedin.com
2018-08-06 15:39:46.974 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Got available container ID (12) for container: Container: [ContainerId:
container_e20_1528615592911_1987_02_000062, NodeId:
lca1-app0596.stg.linkedin.com:1158, NodeHttpAddress:
lca1-app0596.stg.linkedin.com:8042, Resource: <memory:4096, vCores:1>,
Priority: 1, Token: Token { kind: ContainerToken, service: 10.251.166.210:1158
}, ]
2018-08-06 15:39:46.974 [Container Allocator Thread] YarnClusterResourceManager
[INFO] In runContainer in util: fwkPath= ;cmdPath=./__package/;jobLib=
2018-08-06 15:39:46.974 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Container ID 12 using command ./__package//bin/run-container.sh
2018-08-06 15:39:46.975 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Container ID 12 using environment variables:
SAMZA_CONTAINER_ID=12
EXECUTION_ENV_CONTAINER_ID=container_e20_1528615592911_1987_02_000062
SAMZA_COORDINATOR_URL=http://lca1-app1576.stg.linkedin.com:44222/
JAVA_OPTS=-Xmx3072m -Dcom.linkedin.app.name=ad-web-analytics-event-aggregator
-Dcom.linkedin.app.env=ei-lca1
{code}
But due to the issue where the localization phase fails (currently under
investigation) the container fails to start, throwing the following exception
to the `onStartContainerError` callback in `YarnClusterResourceManager`.
{code}
2018-08-06 15:46:15.257
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #6]
YarnClusterResourceManager [ERROR] Container:
container_e20_1528615592911_1987_02_000062 could not start.
java.net.ConnectException: Call From
lca1-app1576.stg.linkedin.com/10.251.174.104 to
lca1-app0596.stg.linkedin.com:1158 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor374.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
at org.apache.hadoop.ipc.Client.call(Client.java:1473)
at org.apache.hadoop.ipc.Client.call(Client.java:1400)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy34.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy35.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:201)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StartContainerTransition.transition(NMClientAsyncImpl.java:377)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StartContainerTransition.transition(NMClientAsyncImpl.java:363)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer.handle(NMClientAsyncImpl.java:498)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$ContainerEventProcessor.run(NMClientAsyncImpl.java:557)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:608)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:706)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:369)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
at org.apache.hadoop.ipc.Client.call(Client.java:1439)
... 22 more
2018-08-06 15:46:15.259
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #6]
YarnClusterResourceManager [INFO] Got an invalid notification for container:
container_e20_1528615592911_1987_02_000062
{code}
Looking at the code in `onStartContainerError`: it seems like we are picking a
container from a Map called ` containersPendingStartup` which doesn’t seem to
have valid containers (quick code search did not show when this map is
populated). This causes these failure callbacks to go unaddressed leaving the
job to have containers that are always pending and the only resolution is to
bounce the job.
was:
I noticed this behavior in the AM today where it fails to allocate a new
container on failure to start. The AM seems to get a callback for a container
failure during container startup and is never rescheduled again by the AM. The
logs seem to suggest that the request was made by the AM to start container
`container_e20_1528615592911_1987_02_000062` as seen below:
{code}
2018-08-06 15:39:46.369 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Received launch request for 12 on hostname lca1-app0596.stg.linkedin.com
2018-08-06 15:39:46.974 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Got available container ID (12) for container: Container: [ContainerId:
container_e20_1528615592911_1987_02_000062, NodeId:
lca1-app0596.stg.linkedin.com:1158, NodeHttpAddress:
lca1-app0596.stg.linkedin.com:8042, Resource: <memory:4096, vCores:1>,
Priority: 1, Token: Token { kind: ContainerToken, service: 10.251.166.210:1158
}, ]
2018-08-06 15:39:46.974 [Container Allocator Thread] YarnClusterResourceManager
[INFO] In runContainer in util: fwkPath= ;cmdPath=./__package/;jobLib=
2018-08-06 15:39:46.974 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Container ID 12 using command ./__package//bin/run-container.sh
2018-08-06 15:39:46.975 [Container Allocator Thread] YarnClusterResourceManager
[INFO] Container ID 12 using environment variables:
SAMZA_CONTAINER_ID=12
EXECUTION_ENV_CONTAINER_ID=container_e20_1528615592911_1987_02_000062
SAMZA_COORDINATOR_URL=http://lca1-app1576.stg.linkedin.com:44222/
JAVA_OPTS=-Xmx3072m -Dcom.linkedin.app.name=ad-web-analytics-event-aggregator
-Dcom.linkedin.app.env=ei-lca1
{code}
But due to the issue where the localization phase fails (currently under
investigation) the container fails to start, throwing the following exception
to the `onStartContainerError` callback in `YarnClusterResourceManager`.
2018-08-06 15:46:15.257
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #6]
YarnClusterResourceManager [ERROR] Container:
container_e20_1528615592911_1987_02_000062 could not start.
java.net.ConnectException: Call From
lca1-app1576.stg.linkedin.com/10.251.174.104 to
lca1-app0596.stg.linkedin.com:1158 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor374.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
at org.apache.hadoop.ipc.Client.call(Client.java:1473)
at org.apache.hadoop.ipc.Client.call(Client.java:1400)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy34.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy35.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:201)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StartContainerTransition.transition(NMClientAsyncImpl.java:377)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StartContainerTransition.transition(NMClientAsyncImpl.java:363)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer.handle(NMClientAsyncImpl.java:498)
at
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$ContainerEventProcessor.run(NMClientAsyncImpl.java:557)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:608)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:706)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:369)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
at org.apache.hadoop.ipc.Client.call(Client.java:1439)
... 22 more
2018-08-06 15:46:15.259
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #6]
YarnClusterResourceManager [INFO] Got an invalid notification for container:
container_e20_1528615592911_1987_02_000062
Looking at the code in `onStartContainerError`: it seems like we are picking a
container from a Map called ` containersPendingStartup` which doesn’t seem to
have valid containers (quick code search did not show when this map is
populated). This causes these failure callbacks to go unaddressed leaving the
job to have containers that are always pending and the only resolution is to
bounce the job.
> Some Samza Containers get stuck at "Starting BrokerProxy for
> hostname:portnum" while others seem to be fine
> -----------------------------------------------------------------------------------------------------------
>
> Key: SAMZA-1371
> URL: https://issues.apache.org/jira/browse/SAMZA-1371
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.11.0, 0.12.0
> Environment: Samza version: 0.11, 0.12
> Kafka version: 0.11.0.0
> Reporter: Ak Ka
> Priority: Blocker
> Attachments: stdout.log, thread_dump.txt
>
>
> I noticed this behavior in the AM today where it fails to allocate a new
> container on failure to start. The AM seems to get a callback for a
> container failure during container startup and is never rescheduled again by
> the AM. The logs seem to suggest that the request was made by the AM to start
> container `container_e20_1528615592911_1987_02_000062` as seen below:
> {code}
> 2018-08-06 15:39:46.369 [Container Allocator Thread]
> YarnClusterResourceManager [INFO] Received launch request for 12 on hostname
> lca1-app0596.stg.linkedin.com
> 2018-08-06 15:39:46.974 [Container Allocator Thread]
> YarnClusterResourceManager [INFO] Got available container ID (12) for
> container: Container: [ContainerId:
> container_e20_1528615592911_1987_02_000062, NodeId:
> lca1-app0596.stg.linkedin.com:1158, NodeHttpAddress:
> lca1-app0596.stg.linkedin.com:8042, Resource: <memory:4096, vCores:1>,
> Priority: 1, Token: Token { kind: ContainerToken, service:
> 10.251.166.210:1158 }, ]
> 2018-08-06 15:39:46.974 [Container Allocator Thread]
> YarnClusterResourceManager [INFO] In runContainer in util: fwkPath=
> ;cmdPath=./__package/;jobLib=
> 2018-08-06 15:39:46.974 [Container Allocator Thread]
> YarnClusterResourceManager [INFO] Container ID 12 using command
> ./__package//bin/run-container.sh
> 2018-08-06 15:39:46.975 [Container Allocator Thread]
> YarnClusterResourceManager [INFO] Container ID 12 using environment variables:
> SAMZA_CONTAINER_ID=12
> EXECUTION_ENV_CONTAINER_ID=container_e20_1528615592911_1987_02_000062
> SAMZA_COORDINATOR_URL=http://lca1-app1576.stg.linkedin.com:44222/
> JAVA_OPTS=-Xmx3072m -Dcom.linkedin.app.name=ad-web-analytics-event-aggregator
> -Dcom.linkedin.app.env=ei-lca1
> {code}
> But due to the issue where the localization phase fails (currently under
> investigation) the container fails to start, throwing the following exception
> to the `onStartContainerError` callback in `YarnClusterResourceManager`.
> {code}
> 2018-08-06 15:46:15.257
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #6]
> YarnClusterResourceManager [ERROR] Container:
> container_e20_1528615592911_1987_02_000062 could not start.
> java.net.ConnectException: Call From
> lca1-app1576.stg.linkedin.com/10.251.174.104 to
> lca1-app0596.stg.linkedin.com:1158 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.reflect.GeneratedConstructorAccessor374.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
> at org.apache.hadoop.ipc.Client.call(Client.java:1473)
> at org.apache.hadoop.ipc.Client.call(Client.java:1400)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy34.startContainers(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy35.startContainers(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:201)
> at
> org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StartContainerTransition.transition(NMClientAsyncImpl.java:377)
> at
> org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StartContainerTransition.transition(NMClientAsyncImpl.java:363)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer.handle(NMClientAsyncImpl.java:498)
> at
> org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$ContainerEventProcessor.run(NMClientAsyncImpl.java:557)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:608)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:706)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:369)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> ... 22 more
> 2018-08-06 15:46:15.259
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #6]
> YarnClusterResourceManager [INFO] Got an invalid notification for container:
> container_e20_1528615592911_1987_02_000062
> {code}
> Looking at the code in `onStartContainerError`: it seems like we are picking
> a container from a Map called ` containersPendingStartup` which doesn’t seem
> to have valid containers (quick code search did not show when this map is
> populated). This causes these failure callbacks to go unaddressed leaving the
> job to have containers that are always pending and the only resolution is to
> bounce the job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
