[jira] [Assigned] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt reassigned MAPREDUCE-6704:
---
Assignee: Bibin A Chundatt
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6796) Remove unused properties from JTConfig.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15627732#comment-15627732 ] Akira Ajisaka commented on MAPREDUCE-6796: -- Looks good to me. I'm thinking we can remove the following properties * mapreduce.jobtracker.http.address * mapreduce.jobtracker.handler.count * mapreduce.jobtracker.restart.recover * mapreduce.jobtracker.taskscheduler * mapreduce.jobtracker.instrumentation * mapreduce.jobtracker.maxtasks.perjob * mapreduce.jobtracker.heartbeats.in.second * mapreduce.jobtracker.persist.jobstatus.hours * mapreduce.jobtracker.persist.jobstatus.dir * mapreduce.jobtracker.retirejobs * mapreduce.jobtracker.retiredjobs.cache.size * mapreduce.jobtracker.taskscheduler.taskalloc.capacitypad * mapreduce.jobtracker.jobinit.threads * mapreduce.jobtracker.expire.trackers.interval * mapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob * mapreduce.jobtracker.hosts.filename * mapreduce.jobtracker.hosts.exclude.filename * mapreduce.jobtracker.jobhistory.lru.cache.size * mapreduce.jobtracker.jobhistory.block.size * mapreduce.jobtracker.jobhistory.completed.location * mapreduce.jobtracker.jobhistory.location * mapreduce.jobtracker.blacklist.average.threshold * mapreduce.jobtracker.tasktracker.maxblacklists * mapreduce.jobtracker.split.metainfo.maxsize * mapreduce.jobtracker.webinterface.trusted because these parameters are used only in ConfigUtil.java. What do you think? > Remove unused properties from JTConfig.java > --- > > Key: MAPREDUCE-6796 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6796 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Akira Ajisaka >Assignee: Haibo Chen > Attachments: mapreduce6796.001.patch > > > JobTracker-side of MAPREDUCE-6794. > There are many unused properties in JTConfig.java. Let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15627640#comment-15627640
]
Akira Ajisaka commented on MAPREDUCE-6704:
--
+1 for #3.
In MAPREDUCE-6702, I wrote a patch to document that we need to set
{{HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME}} to mapreduce.admin.user.env and
yarn.app.mapreduce.am.env, but HADOOP_MAPRED_HOME is not always as same as
HADOOP_COMMON_HOME. I'm thinking it's better to set
yarn.nodemanager.env-whitelist instead of setting these parameters.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported
[ https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15627627#comment-15627627 ] Hudson commented on MAPREDUCE-6765: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10751 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10751/]) MAPREDUCE-6765. MR should not schedule container requests in cases where (rkanter: rev fc2b69eba1c5df59f6175205c27dc7b584df50c0) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java > MR should not schedule container requests in cases where reducer or mapper > containers demand resource larger than the maximum supported > --- > > Key: MAPREDUCE-6765 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: mapreduce6765.001.patch, mapreduce6765.002.patch, > mapreduce6765.003.patch, mapreduce6765.004.patch, mapreduce6765.005.patch > > > When mapper or reducer containers request resource larger than the > maxResourceRequest in the cluster, job is to be killed. In such cases, it is > unnecessary to still schedule container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported
[ https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6765: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 Status: Resolved (was: Patch Available) Thanks [~haibochen]. Committed to trunk and branch-2! > MR should not schedule container requests in cases where reducer or mapper > containers demand resource larger than the maximum supported > --- > > Key: MAPREDUCE-6765 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: mapreduce6765.001.patch, mapreduce6765.002.patch, > mapreduce6765.003.patch, mapreduce6765.004.patch, mapreduce6765.005.patch > > > When mapper or reducer containers request resource larger than the > maxResourceRequest in the cluster, job is to be killed. In such cases, it is > unnecessary to still schedule container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15627323#comment-15627323
]
Hadoop QA commented on MAPREDUCE-6773:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
17s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s {color}
| {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app
generated 2 new + 8 unchanged - 2 fixed = 10 total (was 10) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
The patch generated 1 new + 320 unchanged - 4 fixed = 321 total (was 324)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 54s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 20s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836485/MAPREDUCE-6773-MR-6749.003.patch
|
| JIRA Issue | MAPREDUCE-6773 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 503467350c0d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | MR-6749 / 83b9cdf |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6793/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6793/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6793/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output |
https://builds.apache.o
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Affects Version/s: MR-6749 Status: Patch Available (was: Open) I have updated the patch with changes, and will update with these once we have conclusion about the above and below points. bq. if we plan to use MR_AM_CONTAINER_REUSE_MAX_MAPTASKCONTAINERS (also reduce one), then we need to rename the prefix from yarn.app.mapreduce.am to yarn.app.mapreduce & mapred-default.xml needs to be updated We have already renamed these configs as part of MAPREDUCE-6772 and we are just referring the constants from MRJobConfig.java. Is there anything else you are referring here? bq. JOB_CONTAINER_COMPLETED, is it related to this patch ? should it be in MAPREDUCE-6784? seems like not used anywhere in the current patch. I removed this as part of this patch, will add it with MAPREDUCE-6784 or other jira. bq. ContainerRequestorEvent has almost similar naming to that of ContainerRequestEvent, further was wondering whether we can rename it to ContainerReUseRequestEvent, as its only used by RMContainerReUseRequestor. Currently we are referring ContainerRequestorEvent from RMContainerRequestor and RMContainerReuseRequestor. I think the name RMContainerReUseRequestor would not match properly for RMContainerRequestor. Any suggestions? > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Affects Versions: MR-6749 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-MR-6749.003.patch > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626604#comment-15626604
]
Sangjin Lee commented on MAPREDUCE-6704:
I am +1 for #3. There is the yuck factor of mixing up MR into YARN, but the
issue is significant enough and the change small enough that I think we can
make an exception here.
We can move the discussion to the mailing list to expand the audience.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625447#comment-15625447
]
Daniel Templeton commented on MAPREDUCE-6704:
-
Since this is a rather core issue, I'd love to see some additional opinions
here. [~andrew.wang], [~varun_saxena], [~ajisakaa], [~sjlee0], [~sunilg],
[~Naganarasimha], [~vinodkv], any thoughts? Should we take this to the mailing
list for more visibility?
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6795) Update the document for JobConf#setNumReduceTasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625291#comment-15625291
]
Hudson commented on MAPREDUCE-6795:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10743 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10743/])
MAPREDUCE-6795. Update the document for JobConf#setNumReduceTasks. (aajisaka:
rev 235f39b7c44c5907bd4de063892905577c35cd80)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
> Update the document for JobConf#setNumReduceTasks
> -
>
> Key: MAPREDUCE-6795
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6795
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Yiqun Lin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6795.001.patch, MAPREDUCE-6795.002.patch
>
>
> The following document is for MRv1. We should update the document for
> MapReduce on YARN.
> {code:title=JobConf.java}
>* How many reduces?
>*
>* The right number of reduces seems to be 0.95 or
>* 1.75 multiplied by ( *
>* href="{@docRoot}/../mapred-default.html#mapreduce.tasktracker.reduce.tasks.maximum">
>* mapreduce.tasktracker.reduce.tasks.maximum).
>*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6795) Update the document for JobConf#setNumReduceTasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-6795:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.8.0
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-2, and branch-2.8. Thanks [~linyiqun] for the
contribution!
> Update the document for JobConf#setNumReduceTasks
> -
>
> Key: MAPREDUCE-6795
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6795
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Yiqun Lin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6795.001.patch, MAPREDUCE-6795.002.patch
>
>
> The following document is for MRv1. We should update the document for
> MapReduce on YARN.
> {code:title=JobConf.java}
>* How many reduces?
>*
>* The right number of reduces seems to be 0.95 or
>* 1.75 multiplied by ( *
>* href="{@docRoot}/../mapred-default.html#mapreduce.tasktracker.reduce.tasks.maximum">
>* mapreduce.tasktracker.reduce.tasks.maximum).
>*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6795) Update the document for JobConf#setNumReduceTasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625225#comment-15625225
]
Akira Ajisaka commented on MAPREDUCE-6795:
--
+1, committing this.
> Update the document for JobConf#setNumReduceTasks
> -
>
> Key: MAPREDUCE-6795
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6795
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Yiqun Lin
> Attachments: MAPREDUCE-6795.001.patch, MAPREDUCE-6795.002.patch
>
>
> The following document is for MRv1. We should update the document for
> MapReduce on YARN.
> {code:title=JobConf.java}
>* How many reduces?
>*
>* The right number of reduces seems to be 0.95 or
>* 1.75 multiplied by ( *
>* href="{@docRoot}/../mapred-default.html#mapreduce.tasktracker.reduce.tasks.maximum">
>* mapreduce.tasktracker.reduce.tasks.maximum).
>*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6802) TestKill.testKillJob() fails intermittently on Power
[ https://issues.apache.org/jira/browse/MAPREDUCE-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624587#comment-15624587 ] Yussuf Shaikh commented on MAPREDUCE-6802: -- [~haibochen] Even I think same. I suggest along with waitForState on the job, add a line in the test case to waitForState for the task to be SCHEDULED state. Have tested above workaround on Power and it worked every time. > TestKill.testKillJob() fails intermittently on Power > > > Key: MAPREDUCE-6802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6802 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.7.3 > Environment: # uname -a > Linux pts00452-vm10 3.10.0-327.el7.ppc64le #1 SMP Thu Oct 29 17:31:13 EDT > 2015 ppc64le ppc64le ppc64le GNU/Linux > # cat /etc/redhat-release > Red Hat Enterprise Linux Server release 7.2 (Maipo) >Reporter: Yussuf Shaikh > > Running org.apache.hadoop.mapreduce.v2.app.TestKill > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 9.86 sec <<< > FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestKill > testKillJob(org.apache.hadoop.mapreduce.v2.app.TestKill) Time elapsed: 0.377 > sec <<< FAILURE! > java.lang.AssertionError: Task state not correct expected: but > was: >at org.junit.Assert.fail(Assert.java:88) >at org.junit.Assert.failNotEquals(Assert.java:743) >at org.junit.Assert.assertEquals(Assert.java:118) >at > org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob(TestKill.java:99) > Results : > Failed tests: > TestKill.testKillJob:99 Task state not correct expected: but > was: > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
