[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635184#comment-15635184 ] Hadoop QA commented on MAPREDUCE-6704: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 20s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12837039/MAPREDUCE-6704.0002.patch | | JIRA Issue | MAPREDUCE-6704 | | Optional Tests | asflicense mvnsite | | uname | Linux 3866c2eb9814 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 69dd5fa | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6796/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Container fail to launch for mapred application > --- > > Key: MAPREDUCE-6704 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Blocker > Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch, > MAPREDUCE-6704.0002.patch, container-whitelist-env-wip.patch > > > Container fail to launch for mapred application. > As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set > .After > https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c >{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}} > since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false. > {noformat} > 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with > state FAILED due to: Application application_1462155939310_0004 failed 2 > times due to AM Container for appattempt_1462155939310_0004_02 exited > with exitCode: 1 > Failing this attempt.Diagnostics: Exception from container-launch. > Container id: container_1462155939310_0004_02_01 > Exit code: 1 > Stack trace: ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) > at org.apache.hadoop.util.Shell.run(Shell.java:850) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 1. Last 4096 bytes of stderr : > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; > support was removed in 8.0 > Error: Could not find or load main class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster > Container exited
[jira] [Updated] (MAPREDUCE-6704) Container fail to launch for mapred application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated MAPREDUCE-6704: Attachment: MAPREDUCE-6704.0002.patch Thank you all . Attaching patch based on solution 3 > Container fail to launch for mapred application > --- > > Key: MAPREDUCE-6704 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Blocker > Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch, > MAPREDUCE-6704.0002.patch, container-whitelist-env-wip.patch > > > Container fail to launch for mapred application. > As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set > .After > https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c >{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}} > since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false. > {noformat} > 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with > state FAILED due to: Application application_1462155939310_0004 failed 2 > times due to AM Container for appattempt_1462155939310_0004_02 exited > with exitCode: 1 > Failing this attempt.Diagnostics: Exception from container-launch. > Container id: container_1462155939310_0004_02_01 > Exit code: 1 > Stack trace: ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) > at org.apache.hadoop.util.Shell.run(Shell.java:850) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 1. Last 4096 bytes of stderr : > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; > support was removed in 8.0 > Error: Could not find or load main class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster > Container exited with a non-zero exit code 1. Last 4096 bytes of stderr : > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; > support was removed in 8.0 > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6787: - Status: Open (was: Patch Available) > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: job_1478210774848_0001.xml, mapreduce6787.001.patch, > mapreduce6787.002.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6787: - Attachment: job_1478210774848_0001.xml Thanks for the patch [~haibochen]; this should be useful. I was trying it out, and the downloaded XML file has a bunch of HTML at the end that doesn't belong there (I think it's JHS /conf page). Take a look at the attached jobconf XML file. > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: job_1478210774848_0001.xml, mapreduce6787.001.patch, > mapreduce6787.002.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6807) mapred pipes usage is non-standard and redundant
Grant Sohn created MAPREDUCE-6807: - Summary: mapred pipes usage is non-standard and redundant Key: MAPREDUCE-6807 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6807 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv1 Affects Versions: 2.6.0 Reporter: Grant Sohn Priority: Trivial Running "mapred pipes" returns: {noformat} bin/hadoop pipes [-input ] // Input directory [-output ] // Output directory [-jar // jar filename [-inputformat ] // InputFormat class [-map ] // Java Map class [-partitioner ] // Java Partitioner [-reduce ] // Java Reduce class [-writer ] // Java RecordWriter [-program ] // executable URI [-reduces ] // number of reduces [-lazyOutput ] // createOutputLazily {noformat} No Hadoop CLI uses // comments in the help. Most of them follow the pattern similar to UNIX/Linux man page styling: {noformat} usage: CMD OPTIONS OPTIONS/SUBCMD-BLOCK OPTION/SUBCMD-DEFINITION-BLOCK {noformat} It should be look like this since the comments are really redundant: {noformat} usage: bin/mapred pipes [-input ] [-output ] [-jar ] [-inputformat ] [-map ] [-partitioner ] [-reduce ] [-writer ] [-program ] [-reduces ] [-lazyOutput ] {noformat} then the parameters can be expanded upon in the section below. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6803) MR AppMaster should assign container that is closest to the data
[ https://issues.apache.org/jira/browse/MAPREDUCE-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632657#comment-15632657 ] Hadoop QA commented on MAPREDUCE-6803: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color} | {color:red} MAPREDUCE-6803 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12742477/YARN-3856.002.patch | | JIRA Issue | MAPREDUCE-6803 | | Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6795/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > MR AppMaster should assign container that is closest to the data > > > Key: MAPREDUCE-6803 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6803 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster > Environment: Hadoop cluster with multi-level network hierarchy >Reporter: jaehoon ko > Labels: oct16-medium > Attachments: YARN-3856.001.patch, YARN-3856.002.patch > > > Currently, given a Container request for a host, ResourceManager allocates a > Container with following priorities (RMContainerAllocator.java): > - Requested host > - a host in the same rack as the requested host > - any host > This can lead to a sub-optimal allocation if Hadoop cluster is deployed on > multi-level networked hosts (which is typical). For example, let's suppose a > network architecture with one core switches, two aggregate switches, four ToR > switches, and 8 hosts. Each switch has two downlinks. Rack IDs of hosts are > as follows: > h1, h2: /c/a1/t1 > h3, h4: /c/a1/t2 > h5, h6: /c/a2/t3 > h7, h8: /c/a2/t4 > To allocate a container for data in h1, Hadoop first tries h1 itself, then > h2, then any of h3 ~ h8. Clearly, h3 or h4 are better than h5~h8 in terms of > network distance and bandwidth. However, current implementation choose one > from h3~h8 with equal probabilities. > This limitation is more obvious when considering hadoop clusters deployed on > VM or containers. In this case, only the VMs or containers running in the > same physical host are considered rack local, and actual rack-local hosts are > chosen with same probabilities as far hosts. > The root cause of this limitation is that RMContainerAllocator.java performs > exact matching on rack id to find a rack local host. Alternatively, we can > perform longest-prefix matching to find a closest host. Using the same > network architecture as above, with longest-prefix matching, hosts are > selected with the following priorities: > h1 > h2 > h3 or h4 > h5 or h6 or h7 or h8 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6803) MR AppMaster should assign container that is closest to the data
[ https://issues.apache.org/jira/browse/MAPREDUCE-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632654#comment-15632654 ] Junping Du commented on MAPREDUCE-6803: --- Sorry for replying a bit late as busy in travel previously. From my quick check, most of code is still in YARN side. [~leftnoteasy], May be we should move back to YARN? > MR AppMaster should assign container that is closest to the data > > > Key: MAPREDUCE-6803 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6803 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster > Environment: Hadoop cluster with multi-level network hierarchy >Reporter: jaehoon ko > Labels: oct16-medium > Attachments: YARN-3856.001.patch, YARN-3856.002.patch > > > Currently, given a Container request for a host, ResourceManager allocates a > Container with following priorities (RMContainerAllocator.java): > - Requested host > - a host in the same rack as the requested host > - any host > This can lead to a sub-optimal allocation if Hadoop cluster is deployed on > multi-level networked hosts (which is typical). For example, let's suppose a > network architecture with one core switches, two aggregate switches, four ToR > switches, and 8 hosts. Each switch has two downlinks. Rack IDs of hosts are > as follows: > h1, h2: /c/a1/t1 > h3, h4: /c/a1/t2 > h5, h6: /c/a2/t3 > h7, h8: /c/a2/t4 > To allocate a container for data in h1, Hadoop first tries h1 itself, then > h2, then any of h3 ~ h8. Clearly, h3 or h4 are better than h5~h8 in terms of > network distance and bandwidth. However, current implementation choose one > from h3~h8 with equal probabilities. > This limitation is more obvious when considering hadoop clusters deployed on > VM or containers. In this case, only the VMs or containers running in the > same physical host are considered rack local, and actual rack-local hosts are > chosen with same probabilities as far hosts. > The root cause of this limitation is that RMContainerAllocator.java performs > exact matching on rack id to find a rack local host. Alternatively, we can > perform longest-prefix matching to find a closest host. Using the same > network architecture as above, with longest-prefix matching, hosts are > selected with the following priorities: > h1 > h2 > h3 or h4 > h5 or h6 or h7 or h8 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org