[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689002#comment-15689002
]
Bibin A Chundatt edited comment on MAPREDUCE-6704 at 11/23/16 5:36 AM:
---
[~tangzhankun] thank you for providing test result. YARN-5877 is added to track
the same.
[~templedf] As per
[discussion|https://issues.apache.org/jira/browse/MAPREDUCE-6704?focusedCommentId=15596876=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15596876]
we had identified that Solution 3 is best fit. But for docker to run by
default we should support change YARN-5877 (Currently the vars to be fetched
from env for docker is fixed we should make it configurable) .[Comment
|https://issues.apache.org/jira/browse/MAPREDUCE-6704?focusedCommentId=15661038=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15661038]
explains in details why we should support.
So combination of both would fix completely.
was (Author: bibinchundatt):
[~tangzhankun] thank you for providing test result. YARN-5877 is added to track
the same.
[~templedf] As per
[discussion|https://issues.apache.org/jira/browse/MAPREDUCE-6704?focusedCommentId=15596876=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15596876]
we had identified that Solution 3 is best fit. But for docker to run by
default we should support change YARN-5877 (Currently the vars to be fetched
from env for docker is fixed we should make it configurable)
.[comment2|https://issues.apache.org/jira/browse/MAPREDUCE-6704?focusedCommentId=15661038=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15661038]
explains in details why we should support.
So combination of both will would fix completely.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MR-6704-branch2.8.tar.gz,
> MR-6704-trunk-tempPatch.tar.gz, MR-6704-trunk.tar.gz, SingleCluster.html,
> container-whitelist-env-wip.patch, temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662943#comment-15662943
]
Bibin A Chundatt edited comment on MAPREDUCE-6704 at 11/14/16 6:45 AM:
---
[~tangzhankun] could be related docker mount , YARN-5765 or 2.7 to hadoop.3.0
compatability Sorry i am new to docker so unaware
[~templedf] Added YARN-5877 to track white list case.
was (Author: bibinchundatt):
[~tangzhankun] could be related docker mount , YARN-5765, Sorry i am new to
docker
[~templedf] Added YARN-5877 to track white list case.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MR-6704-branch2.8.tar.gz,
> MR-6704-trunk-tempPatch.tar.gz, MR-6704-trunk.tar.gz, SingleCluster.html,
> container-whitelist-env-wip.patch, temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662513#comment-15662513
]
Zhankun Tang edited comment on MAPREDUCE-6704 at 11/14/16 1:57 AM:
---
[~bibinchundatt], The temp.patch corrects the HADOOP_MAPRED_HOME variable and
the MRAppMaster was found successfully. But then the AM fails on token version
check. Please check the container logs uploaded(MR-6704-trunk-tempPatch.tar.gz).
{noformat}
2016-11-14 04:43:47,660 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.io.IOException: Exception reading
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1479115665737_0004/container_1479115665737_0004_02_01/container_tokens
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:198)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:816)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1495)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.io.IOException: Unknown version 1 in token storage.
at
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:216)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:195)
... 5 more
2016-11-14 04:43:47,662 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1
{noformat}
was (Author: tangzhankun):
[~bibinchundatt], The temp.patch corrects the HADOOP_MAPRED_HOME variable and
the MRAppMaster was found successfully. But then the AM fails on token version
check. Please check the container logs(MR-6704-trunk-tempPatch.tar.gz).
{noformat}
2016-11-14 04:43:47,660 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.io.IOException: Exception reading
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1479115665737_0004/container_1479115665737_0004_02_01/container_tokens
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:198)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:816)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1495)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.io.IOException: Unknown version 1 in token storage.
at
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:216)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:195)
... 5 more
2016-11-14 04:43:47,662 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1
{noformat}
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MR-6704-branch2.8.tar.gz,
> MR-6704-trunk-tempPatch.tar.gz, MR-6704-trunk.tar.gz, SingleCluster.html,
> container-whitelist-env-wip.patch, temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662513#comment-15662513
]
Zhankun Tang edited comment on MAPREDUCE-6704 at 11/14/16 1:56 AM:
---
[~bibinchundatt], The temp.patch corrects the HADOOP_MAPRED_HOME variable and
the MRAppMaster was found successfully. But then the AM fails on token version
check. Please check the container logs(MR-6704-trunk-tempPatch.tar.gz).
{noformat}
2016-11-14 04:43:47,660 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.io.IOException: Exception reading
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1479115665737_0004/container_1479115665737_0004_02_01/container_tokens
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:198)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:816)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1495)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.io.IOException: Unknown version 1 in token storage.
at
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:216)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:195)
... 5 more
2016-11-14 04:43:47,662 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1
{noformat}
was (Author: tangzhankun):
[~bibinchundatt], The temp.patch corrects the HADOOP_MAPRED_HOME variable and
the MRAppMaster was found successfully. But then the AM fails on token version
check. Please check the container logs.
{noformat}
2016-11-14 04:43:47,660 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.io.IOException: Exception reading
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1479115665737_0004/container_1479115665737_0004_02_01/container_tokens
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:198)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:816)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1495)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.io.IOException: Unknown version 1 in token storage.
at
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:216)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:195)
... 5 more
2016-11-14 04:43:47,662 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1
{noformat}
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MR-6704-branch2.8.tar.gz,
> MR-6704-trunk-tempPatch.tar.gz, MR-6704-trunk.tar.gz, SingleCluster.html,
> container-whitelist-env-wip.patch, temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
>
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662513#comment-15662513
]
Zhankun Tang edited comment on MAPREDUCE-6704 at 11/14/16 1:54 AM:
---
[~bibinchundatt], The temp.patch corrects the HADOOP_MAPRED_HOME variable and
the MRAppMaster was found successfully. But then the AM fails on token version
check. Please check the container logs.
{noformat}
2016-11-14 04:43:47,660 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.io.IOException: Exception reading
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1479115665737_0004/container_1479115665737_0004_02_01/container_tokens
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:198)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:816)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1495)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.io.IOException: Unknown version 1 in token storage.
at
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:216)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:195)
... 5 more
2016-11-14 04:43:47,662 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1
{noformat}
was (Author: tangzhankun):
[~bibinchundatt], The temp.patch corrects the HADOOP_MAPRED_HOME variable and
the MRAppMaster was found successfully. But then the AM fails on token version
check. Please check the container logs.
```
2016-11-14 04:43:47,660 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.io.IOException: Exception reading
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1479115665737_0004/container_1479115665737_0004_02_01/container_tokens
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:198)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:816)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1495)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.io.IOException: Unknown version 1 in token storage.
at
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:216)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:195)
... 5 more
2016-11-14 04:43:47,662 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1
```
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MR-6704-branch2.8.tar.gz,
> MR-6704-trunk-tempPatch.tar.gz, MR-6704-trunk.tar.gz, SingleCluster.html,
> container-whitelist-env-wip.patch, temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
>
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662513#comment-15662513
]
Zhankun Tang edited comment on MAPREDUCE-6704 at 11/14/16 1:53 AM:
---
[~bibinchundatt], The temp.patch corrects the HADOOP_MAPRED_HOME variable and
the MRAppMaster was found successfully. But then the AM fails on token version
check. Please check the container logs.
```
2016-11-14 04:43:47,660 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.io.IOException: Exception reading
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1479115665737_0004/container_1479115665737_0004_02_01/container_tokens
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:198)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:816)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1495)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.io.IOException: Unknown version 1 in token storage.
at
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:216)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:195)
... 5 more
2016-11-14 04:43:47,662 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1
```
was (Author: tangzhankun):
[~bibinchundatt], The temp.patch corrects the HADOOP_MAPRED_HOME variable and
the MRAppMaster was found successfully. But then the AM fails on token version
check. Please check the container logs.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MR-6704-branch2.8.tar.gz,
> MR-6704-trunk-tempPatch.tar.gz, MR-6704-trunk.tar.gz, SingleCluster.html,
> container-whitelist-env-wip.patch, temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654304#comment-15654304
]
Bibin A Chundatt edited comment on MAPREDUCE-6704 at 11/10/16 3:18 PM:
---
[~templedf]
I was thinking of handling through mailing list and analyze the logs . We will
other any other env needs to specified
[~tangzhankun]
As Daniel mentioned after configuring
{{yarn.nodemanager.delete.debug-delay-sec}} please attach the following
# launch script
# node manager logs during docker run.
was (Author: bibinchundatt):
[~templedf]
Was thinking of handling though mailing list and analyze the logs . Will other
any other env needs specified.
[~tangzhankun]
As Daniel mentioned after configuring
{{yarn.nodemanager.delete.debug-delay-sec}} please attach the following
# launch script
# node manager logs during docker run.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, SingleCluster.html,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15617802#comment-15617802
]
Bibin A Chundatt edited comment on MAPREDUCE-6704 at 10/29/16 10:56 AM:
Thank you [~templedf] and [~rkanter] for inputs
#3 since for upgrade case when we have running apps and havnt used
distributed cache.
Need to modify only one config during upgrade and everything works as old.
Another case can happen when old clients try to submit apps too rt?
{quote}
client doesn't have the same paths as the NM hosts, it won't work.
{quote}
[~templedf] Thank you for bring this scenarios too had missed out.
Will do document update and upload patch soon.
was (Author: bibinchundatt):
Thank you [~templedf] and [~rkanter] for inputs
#3 since for upgrade case when already application we have running apps and
havnt used distributed cache.
Need to modify only one config during upgrade and everything works as old.
Another case can happen when old clients try to submit apps too rt?
{quote}
client doesn't have the same paths as the NM hosts, it won't work.
{quote}
[~templedf] Thank you for bring this scenarios too had missed out.
Will do document update and upload patch soon.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612421#comment-15612421
]
Daniel Templeton edited comment on MAPREDUCE-6704 at 10/28/16 1:39 PM:
---
+1 to #3, but *making it the default*, not just documenting it. I agree with
[~aw] that a little pollution is OK if it makes the out-of-box experience
significantly better. And I don't think spilling over an environment variable
is so bad anyway.
#1 has several issues. In addition to what was already stated, the vars are
evaluated on the client, so if the client doesn't have the same paths as the NM
hosts, it won't work.
#2 is a code-level pollution, which is a more serious thing than #3.
#4 doesn't sound like the right idea to me. Users don't need to control the
whitelisting on a per-job basis, so it's solving a problem that doesn't exist.
It also has some security implications that need to be carefully considered.
was (Author: templedf):
+1 to #3. I agree with [~aw] that a little pollution is OK if it makes the
out-of-box experience significantly better. And I don't think spilling over an
environment variable is so bad anyway.
#1 has several issues. In addition to what was already stated, the vars are
evaluated on the client, so if the client doesn't have the same paths as the NM
hosts, it won't work.
#2 is a code-level pollution, which is a more serious thing than #3.
#4 doesn't sound like the right idea to me. Users don't need to control the
whitelisting on a per-job basis, so it's solving a problem that doesn't exist.
It also has some security implications that need to be carefully considered.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612421#comment-15612421
]
Daniel Templeton edited comment on MAPREDUCE-6704 at 10/27/16 4:30 PM:
---
+1 to #3. I agree with [~aw] that a little pollution is OK if it makes the
out-of-box experience significantly better. And I don't think spilling over an
environment variable is so bad anyway.
#1 has several issues. In addition to what was already stated, the vars are
evaluated on the client, so if the client doesn't have the same paths as the NM
hosts, it won't work.
#2 is a code-level pollution, which is a more serious thing than #3.
#4 doesn't sound like the right idea to me. Users don't need to control the
whitelisting on a per-job basis, so it's solving a problem that doesn't exist.
It also has some security implications that need to be carefully considered.
was (Author: templedf):
+1 to #3. A agree with [~aw] that a little pollution is OK if it makes the
out-of-box experience significantly better. And I don't think spilling over an
environment variable is so bad anyway.
#1 has several issues. In addition to what was already stated, the vars are
evaluated on the client, so if the client doesn't have the same paths as the NM
hosts, it won't work.
#2 is a code-level pollution, which is a more serious thing than #3.
#4 doesn't sound like the right idea to me. Users don't need to control the
whitelisting on a per-job basis, so it's solving a problem that doesn't exist.
It also has some security implications that need to be carefully considered.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509009#comment-15509009
]
Bibin A Chundatt edited comment on MAPREDUCE-6704 at 9/21/16 7:10 AM:
--
[~tangzhankun]
In nodemanager we can add the below config. Last time i checked below solution
was working fine.
yarn.nodemanager.env-whitelist=
"JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME"
was (Author: bibinchundatt):
[~tangzhankun]
In nodemanager we can add we can add the below config last time i checked below
solution was working fine.
yarn.nodemanager.env-whitelist=
"JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME"
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508908#comment-15508908
]
Zhankun Tang edited comment on MAPREDUCE-6704 at 9/21/16 6:45 AM:
--
I encountered this issue too. Quite shocked that commit "9d4d30243b0fc" get
merged without basic MRv2 test and even no YARN JIRA for it!
And it's strange that I still fail to run MRv2 application as well as
Dockerized MRv2 application after tried the temporary solution mentioned by
Bibin A Chundatt. YARN patch also not works for me.
was (Author: tangzhankun):
I encountered this issue too. Quite shocked that commit "9d4d30243b0fc" get
merged without basic MRv2 test and even no YARN JIRA for it!
And it's strange that I still fail to run MRv2 application as well as
Dockerized MRv2 application after tried the temporary solution mentioned by
Bibin A Chundatt.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410998#comment-15410998
]
Bibin A Chundatt edited comment on MAPREDUCE-6704 at 8/8/16 4:39 AM:
-
[~aw]/[~ajisakaa]
Sorry for the delay.
As already mentioned the {{HADOOP_MAPRED_HOME}} is not part of env since not
inherited to launcher.Adding
{code}
yarn.nodemanager.env-whitelist="JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME"
{code}
in yarn-site.xml of NM does solve the problem. But since its a mapreduce
property its not correct to add to the whitelist of yarn .
IIUU {{HADOOP_MAPRED_HOME}} for default installation will be same as
{{HADOOP_COMMON_HOME}} .
For layout change we can configure explicity mapred.child.env paths.
Till jira is fixed ,to make mapreduce job run add params as below
{code}
-Dmapred.child.env="HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}"
-Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}"
{code}
Open to any other solution.
was (Author: bibinchundatt):
[~aw]/[~ajisakaa]
Sorry for the delay.
As already mentioned the {{HADOOP_MAPRED_HOME}} is not part of env since not
inherited to launcher.Adding
{code}
yarn.nodemanager.env-whitelist="JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME"
{code}
in yarn-site.xml of NM does solve the problem. But since its a mapreduce
property its not correct to add to the whitelist of yarn .
IIUU {{HADOOP_MAPRED_HOME}} for default installation will be same as
{{HADOOP_COMMON_HOME}} .
For layout change we can configure explicity mapred.child.env paths.
Will jira is fixed ,to make mapreduce job run add params as below
{code}
-Dmapred.child.env="HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}"
-Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}"
{code}
Open to any other solution.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
