[jira] [Commented] (MAPREDUCE-5705) mapreduce.task.io.sort.mb hardcoded cap at 2047
[ https://issues.apache.org/jira/browse/MAPREDUCE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225221#comment-15225221 ] Karthik Kambatla commented on MAPREDUCE-5705: - Isn't this a duplicate of MAPREDUCE-5028? > mapreduce.task.io.sort.mb hardcoded cap at 2047 > --- > > Key: MAPREDUCE-5705 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5705 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 > Environment: Multinode Dell XD720 cluster Centos6 running HDP2 >Reporter: Joseph Niemiec > > mapreduce.task.io.sort.mb is hardcoded to not allow values larger then 2047. > If you enter a value larger then this the map tasks will always crash at this > line - > https://github.com/apache/hadoop-mapreduce/blob/HDFS-641/src/java/org/apache/hadoop/mapred/MapTask.java?source=cc#L746 > The nodes at dev site have over 380 GB of Ram each, we are not able to make > the best use of large mappers (15GB mappers) because of the hardcoded buffer > max. Is there a reason this value has been hardcoded? > -- > Also validated on my dev VM. Indeed setting io.sort.mb to 2047 works but 2048 > fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6334) Fetcher#copyMapOutput is leaking usedMemory upon IOException during InMemoryMapOutput shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224413#comment-15224413 ] Eric Payne commented on MAPREDUCE-6334: --- [~vishal.rajan], what version of Hadoop are you running? > Fetcher#copyMapOutput is leaking usedMemory upon IOException during > InMemoryMapOutput shuffle handler > - > > Key: MAPREDUCE-6334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 2.7.1, 2.6.2 > > Attachments: MAPREDUCE-6334.001.patch, MAPREDUCE-6334.002.patch > > > We are seeing this happen when > - an NM's disk goes bad during the creation of map output(s) > - the reducer's fetcher can read the shuffle header and reserve the memory > - but gets an IOException when trying to shuffle for InMemoryMapOutput > - shuffle fetch retry is enabled -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6643) org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224291#comment-15224291 ] Andras Bokor commented on MAPREDUCE-6643: - When seed is 855631073 it also fails. > org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed > --- > > Key: MAPREDUCE-6643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6643 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Haibo Chen > > Unit test TestTextInputFormat.testSplitableCodecs() failed when the seed is > 1313094493. > Stacktrace > java.lang.AssertionError: Key in multiple partitions. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertFalse(Assert.java:64) > at > org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs(TestTextInputFormat.java:223) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5705) mapreduce.task.io.sort.mb hardcoded cap at 2047
[ https://issues.apache.org/jira/browse/MAPREDUCE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224234#comment-15224234 ] Junping Du commented on MAPREDUCE-5705: --- MAPREDUCE-2308 is a very old JIRA for MRv1 age. Let's reopen this and fix it in 2.x. > mapreduce.task.io.sort.mb hardcoded cap at 2047 > --- > > Key: MAPREDUCE-5705 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5705 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 > Environment: Multinode Dell XD720 cluster Centos6 running HDP2 >Reporter: Joseph Niemiec > > mapreduce.task.io.sort.mb is hardcoded to not allow values larger then 2047. > If you enter a value larger then this the map tasks will always crash at this > line - > https://github.com/apache/hadoop-mapreduce/blob/HDFS-641/src/java/org/apache/hadoop/mapred/MapTask.java?source=cc#L746 > The nodes at dev site have over 380 GB of Ram each, we are not able to make > the best use of large mappers (15GB mappers) because of the hardcoded buffer > max. Is there a reason this value has been hardcoded? > -- > Also validated on my dev VM. Indeed setting io.sort.mb to 2047 works but 2048 > fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MAPREDUCE-5705) mapreduce.task.io.sort.mb hardcoded cap at 2047
[ https://issues.apache.org/jira/browse/MAPREDUCE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du reopened MAPREDUCE-5705: --- > mapreduce.task.io.sort.mb hardcoded cap at 2047 > --- > > Key: MAPREDUCE-5705 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5705 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 > Environment: Multinode Dell XD720 cluster Centos6 running HDP2 >Reporter: Joseph Niemiec > > mapreduce.task.io.sort.mb is hardcoded to not allow values larger then 2047. > If you enter a value larger then this the map tasks will always crash at this > line - > https://github.com/apache/hadoop-mapreduce/blob/HDFS-641/src/java/org/apache/hadoop/mapred/MapTask.java?source=cc#L746 > The nodes at dev site have over 380 GB of Ram each, we are not able to make > the best use of large mappers (15GB mappers) because of the hardcoded buffer > max. Is there a reason this value has been hardcoded? > -- > Also validated on my dev VM. Indeed setting io.sort.mb to 2047 works but 2048 > fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223845#comment-15223845 ] Kai Sasaki commented on MAPREDUCE-6607: --- [~ozawa] I rebased the patch on trunk. Could you check it again please? > .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or > mapreduce.task.files.preserve.filepattern are set > --- > > Key: MAPREDUCE-6607 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.7.1 >Reporter: Maysam Yabandeh >Assignee: Kai Sasaki >Priority: Minor > Attachments: MAPREDUCE-6607.01.patch, MAPREDUCE-6607.02.patch, > MAPREDUCE-6607.03.patch, MAPREDUCE-6607.04.patch > > > if either of the following configs are set, then .staging dir is not cleaned > up: > * mapreduce.task.files.preserve.failedtask > * mapreduce.task.files.preserve.filepattern > The former was supposed to keep only .staging of failed tasks and the latter > was supposed to be used only if that task name matches against the specified > regular expression. > {code} > protected boolean keepJobFiles(JobConf conf) { > return (conf.getKeepTaskFilesPattern() != null || conf > .getKeepFailedTaskFiles()); > } > {code} > {code} > public void cleanupStagingDir() throws IOException { > /* make sure we clean the staging files */ > String jobTempDir = null; > FileSystem fs = getFileSystem(getConfig()); > try { > if (!keepJobFiles(new JobConf(getConfig( { > jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR); > if (jobTempDir == null) { > LOG.warn("Job Staging directory is null"); > return; > } > Path jobTempDirPath = new Path(jobTempDir); > LOG.info("Deleting staging directory " + > FileSystem.getDefaultUri(getConfig()) + > " " + jobTempDir); > fs.delete(jobTempDirPath, true); > } > } catch(IOException io) { > LOG.error("Failed to cleanup staging dir " + jobTempDir, io); > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223832#comment-15223832 ] Hadoop QA commented on MAPREDUCE-6607: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 33s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 42s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 18s {color} | {color:green} hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 54s {color} | {color:green} hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 33m 17s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:fbe3e86 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12796790/MAPREDUCE-6607.04.patch | | JIRA Issue | MAPREDUCE-6607 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 0665a53a5659 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 1e6f929 | | Default Java | 1.7.0_95 | | Multi-JDK
[jira] [Updated] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated MAPREDUCE-6607: -- Attachment: MAPREDUCE-6607.04.patch > .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or > mapreduce.task.files.preserve.filepattern are set > --- > > Key: MAPREDUCE-6607 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.7.1 >Reporter: Maysam Yabandeh >Assignee: Kai Sasaki >Priority: Minor > Attachments: MAPREDUCE-6607.01.patch, MAPREDUCE-6607.02.patch, > MAPREDUCE-6607.03.patch, MAPREDUCE-6607.04.patch > > > if either of the following configs are set, then .staging dir is not cleaned > up: > * mapreduce.task.files.preserve.failedtask > * mapreduce.task.files.preserve.filepattern > The former was supposed to keep only .staging of failed tasks and the latter > was supposed to be used only if that task name matches against the specified > regular expression. > {code} > protected boolean keepJobFiles(JobConf conf) { > return (conf.getKeepTaskFilesPattern() != null || conf > .getKeepFailedTaskFiles()); > } > {code} > {code} > public void cleanupStagingDir() throws IOException { > /* make sure we clean the staging files */ > String jobTempDir = null; > FileSystem fs = getFileSystem(getConfig()); > try { > if (!keepJobFiles(new JobConf(getConfig( { > jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR); > if (jobTempDir == null) { > LOG.warn("Job Staging directory is null"); > return; > } > Path jobTempDirPath = new Path(jobTempDir); > LOG.info("Deleting staging directory " + > FileSystem.getDefaultUri(getConfig()) + > " " + jobTempDir); > fs.delete(jobTempDirPath, true); > } > } catch(IOException io) { > LOG.error("Failed to cleanup staging dir " + jobTempDir, io); > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6668) AM of no uber job does not need local resources.
KWON BYUNGCHANG created MAPREDUCE-6668: -- Summary: AM of no uber job does not need local resources. Key: MAPREDUCE-6668 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6668 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Reporter: KWON BYUNGCHANG Assignee: KWON BYUNGCHANG Priority: Minor if specify {{-libjar}} {{-files}} {{-archive}} on command line or set below properties {code} mapreduce.job.cache.archives mapreduce.job.cache.files.filesizes {code} MapReduce framework will configure {{LocalResources}} for map/reduce container. also configure for ApplicationMaster. if job is not uber, ApplicationMaster does not need {{LocalResources}}. Deploying localResource for AM will cause slowly launching AM. I make a proposal: if job is not uber, map-reduce client does not configure localResource for AM. obviously must configure for map/reduce container. -- This message was sent by Atlassian JIRA (v6.3.4#6332)