[
https://issues.apache.org/jira/browse/YARN-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-3112:
-----------------------------------
Description:
This error is very similar to YARN-1795, YARN-1839, but i have check the
solution of those jira, the patches are already included in my version. I think
this error is caused by the different NMTokens between old and new appattempts.
New AM has inherited the old tokens from previous AM according to my
configuration (keepContainers=true), so the token for new containers are
replaced by the old one in the NMTokenCache.
{noformat}
206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container
launch failed for container_1422546145900_0001_02_000002 :
org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent
for ixk02:47625
207 › at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt
ocolProxy.java:256)
208 › at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtoc
olProxy.java:246)
209 › at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132)
210 › at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401)
211 › at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
212 › at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367)
213 › at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
214 › at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
215 › at java.lang.Thread.run(Thread.java:722)
{noformat}
was:
This error is very similar to YARN-1795, YARN-1839, but i have check the
solution of those jira, the patches are already included in my version. I think
this error is caused by the different NMTokens between old and new appattempts.
New AM has inherited the old tokens from previous AM according to my
configuration (keepContainers=true), so the token for new containers are
replaced by the old one in the NMTokenCache.
206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container
launch failed for container_1422546145900_0001_02_000002 :
org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent
for ixk02:47625
207 › at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt
ocolProxy.java:256)
208 › at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtoc
olProxy.java:246)
209 › at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132)
210 › at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401)
211 › at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
212 › at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367)
213 › at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
214 › at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
215 › at java.lang.Thread.run(Thread.java:722)
> AM restart and keep containers from previous attempts, then new container
> launch failed
> ---------------------------------------------------------------------------------------
>
> Key: YARN-3112
> URL: https://issues.apache.org/jira/browse/YARN-3112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications, resourcemanager
> Affects Versions: 2.6.0
> Environment: in real linux cluster
> Reporter: Jack Chen
>
> This error is very similar to YARN-1795, YARN-1839, but i have check the
> solution of those jira, the patches are already included in my version. I
> think this error is caused by the different NMTokens between old and new
> appattempts. New AM has inherited the old tokens from previous AM according
> to my configuration (keepContainers=true), so the token for new containers
> are replaced by the old one in the NMTokenCache.
> {noformat}
> 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container
> launch failed for container_1422546145900_0001_02_000002 :
> org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent
> for ixk02:47625
> 207 › at
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt
> ocolProxy.java:256)
> 208 › at
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtoc
> olProxy.java:246)
> 209 › at
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132)
> 210 › at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401)
> 211 › at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
> 212 › at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367)
> 213 › at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 214 › at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 215 › at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
