[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199540#comment-16199540 ] Sangjin Lee commented on MAPREDUCE-5951: +1. Thanks for the great work and taking it to completion [~ctrezzo]! > Add support for the YARN Shared Cache > - > > Key: MAPREDUCE-5951 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5951-Overview.001.pdf, > MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, > MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, > MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch, > MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, > MAPREDUCE-5951-trunk-v15.patch, MAPREDUCE-5951-trunk-v2.patch, > MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, > MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, > MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, > MAPREDUCE-5951-trunk-v9.patch, MAPREDUCE-5951-trunk.016.patch, > MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch, > MAPREDUCE-5951-trunk.019.patch > > > Implement the necessary changes so that the MapReduce application can > leverage the new YARN shared cache (i.e. YARN-1492). > Specifically, allow per-job configuration so that MapReduce jobs can specify > which set of resources they would like to cache (i.e. jobjar, libjars, > archives, files). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[ https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957590#comment-15957590 ] Sangjin Lee commented on MAPREDUCE-6846: Thanks [~dan...@cloudera.com]! Would you do the honors of committing this? > Fragments specified for libjar paths are not handled correctly > -- > > Key: MAPREDUCE-6846 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2 >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Attachments: MAPREDUCE-6846-trunk.001.patch, > MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch, > MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch, > MAPREDUCE-6846-trunk.006.patch > > > If a user specifies a fragment for a libjars path via generic options parser, > the client crashes with a FileNotFoundException: > {noformat} > java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt > does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314) > at > org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105) > at > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362) > at > org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306) > at > org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at > org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:239) > at org.apache.hadoop.util.RunJar.main(RunJar.java:153) > {noformat} > This is actually inconsistent with the behavior for files and archives. Here > is a table showing the current behavior for each type of path and resource: > | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute > path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e. > test.txt#frag.txt) || > || -libjars | FileNotFound | FileNotFound|FileNotFound| > || -files | (/) | (/) | (/) | > || -archives | (/) | (/) | (/) | -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[ https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954409#comment-15954409 ] Sangjin Lee commented on MAPREDUCE-6846: The latest patch LGTM. [~dan...@cloudera.com] what do you think? > Fragments specified for libjar paths are not handled correctly > -- > > Key: MAPREDUCE-6846 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2 >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Attachments: MAPREDUCE-6846-trunk.001.patch, > MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch, > MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch, > MAPREDUCE-6846-trunk.006.patch > > > If a user specifies a fragment for a libjars path via generic options parser, > the client crashes with a FileNotFoundException: > {noformat} > java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt > does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314) > at > org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105) > at > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362) > at > org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306) > at > org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at > org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:239) > at org.apache.hadoop.util.RunJar.main(RunJar.java:153) > {noformat} > This is actually inconsistent with the behavior for files and archives. Here > is a table showing the current behavior for each type of path and resource: > | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute > path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e. > test.txt#frag.txt) || > || -libjars | FileNotFound | FileNotFound|FileNotFound| > || -files | (/) | (/) | (/) | > || -archives | (/) | (/) | (/) | -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee reassigned MAPREDUCE-6732: -- Assignee: Vrushali C (was: Sangjin Lee) > mapreduce tasks for YARN Timeline Service v.2: alpha 2 > -- > > Key: MAPREDUCE-6732 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Sangjin Lee >Assignee: Vrushali C > > This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline > Service v.2 alpha 2. > This is developed on feature branches: {{YARN-5355}} for the trunk-based > development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any > subtask work on this JIRA will be committed to those 2 branches. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[ https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922870#comment-15922870 ] Sangjin Lee commented on MAPREDUCE-6846: bq. I took a look through various branches and see that it has been broken at least back to branch-2.0.0-alpha. It seems that this has been broken for a very long time. That's interesting. We saw that terasort uses fragments. But I don't think it has been broken that long? > Fragments specified for libjar paths are not handled correctly > -- > > Key: MAPREDUCE-6846 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2 >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Attachments: MAPREDUCE-6846-trunk.001.patch > > > If a user specifies a fragment for a libjars path via generic options parser, > the client crashes with a FileNotFoundException: > {noformat} > java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt > does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314) > at > org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105) > at > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362) > at > org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306) > at > org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at > org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:239) > at org.apache.hadoop.util.RunJar.main(RunJar.java:153) > {noformat} > This is actually inconsistent with the behavior for files and archives. Here > is a table showing the current behavior for each type of path and resource: > | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute > path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e. > test.txt#frag.txt) || > || -libjars | FileNotFound | FileNotFound|FileNotFound| > || -files | (/) | (/) | (/) | > || -archives | (/) | (/) | (/) | -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[ https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922622#comment-15922622 ] Sangjin Lee commented on MAPREDUCE-6846: Thanks [~ctrezzo] for your patch! I'm generally fine with the approach taken in this patch. Just curious, was this always broken, or is it a regression? If the latter, could you add the JIRA that introduced the regression? (JobResourceUploader.java) - l.169: {{getPathURI()}} will always result in a URL with a fragment even if the original path did not have the fragment. This is not incorrect, but this would have an effect of bloating the distributed cache entries in the configuration which can be sizable in cases. Can we not add the fragment if the fragment is null and the original URL does not have the fragment either? - l.171: nit: it can simply be {{foundFragment = tmpURI.getFragment() != null}} - I'm wondering, would it lead to simpler code if we iterate over the list once to determine whether there is a fragment and iterate again to do the population of the distributed cache? It might be slightly more expensive, but may lead to code that's easier to understand/maintain. Thoughts? > Fragments specified for libjar paths are not handled correctly > -- > > Key: MAPREDUCE-6846 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.3, 3.0.0-alpha2 >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Attachments: MAPREDUCE-6846-trunk.001.patch > > > If a user specifies a fragment for a libjars path via generic options parser, > the client crashes with a FileNotFoundException: > {noformat} > java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt > does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314) > at > org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105) > at > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362) > at > org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306) > at > org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at > org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:239) > at org.apache.hadoop.util.RunJar.main(RunJar.java:153) > {noformat} > This is actually inconsistent wi
[jira] [Commented] (MAPREDUCE-6855) Specify charset when create String in CredentialsTestJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894642#comment-15894642 ] Sangjin Lee commented on MAPREDUCE-6855: Thanks. > Specify charset when create String in CredentialsTestJob > > > Key: MAPREDUCE-6855 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6855 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Akira Ajisaka >Priority: Minor > Labels: newbie > > {code} > String secretValueStr = new String (secretValue); > {code} > should be > {code} > String secretValueStr = new String(secretValue, StandardCharsets.UTF_8); > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6855) Specify charset when create String in CredentialsTestJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894615#comment-15894615 ] Sangjin Lee commented on MAPREDUCE-6855: It would be great if we can catch things like this with a findbugs rule. Do you know if that is straightforward? > Specify charset when create String in CredentialsTestJob > > > Key: MAPREDUCE-6855 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6855 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Akira Ajisaka >Priority: Minor > Labels: newbie > > {code} > String secretValueStr = new String (secretValue); > {code} > should be > {code} > String secretValueStr = new String(secretValue, StandardCharsets.UTF_8); > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5951) Add support for the YARN Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-5951: --- Status: Open (was: Patch Available) Just to reflect the current status (waiting for an updated patch). Let me know if that is not accurate. Thanks! > Add support for the YARN Shared Cache > - > > Key: MAPREDUCE-5951 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5951-Overview.001.pdf, > MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, > MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, > MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, > MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, > MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, > MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, > MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, > MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, > MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, > MAPREDUCE-5951-trunk-v9.patch > > > Implement the necessary changes so that the MapReduce application can > leverage the new YARN shared cache (i.e. YARN-1492). > Specifically, allow per-job configuration so that MapReduce jobs can specify > which set of resources they would like to cache (i.e. jobjar, libjars, > archives, files). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[ https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860182#comment-15860182 ] Sangjin Lee commented on MAPREDUCE-6846: Thanks for finding this issue [~ctrezzo]. Personally I would prefer option #1... > Fragments specified for libjar paths are not handled correctly > -- > > Key: MAPREDUCE-6846 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.3, 3.0.0-alpha2 >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > > If a user specifies a fragment for a libjars path via generic options parser, > the client crashes with a FileNotFoundException: > {noformat} > java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt > does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314) > at > org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154) > at > org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105) > at > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362) > at > org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306) > at > org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at > org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:239) > at org.apache.hadoop.util.RunJar.main(RunJar.java:153) > {noformat} > This is actually inconsistent with the behavior for files and archives. Here > is a table showing the current behavior for each type of path and resource: > | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute > path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e. > test.txt#frag.txt) || > || -libjars | FileNotFound | FileNotFound|FileNotFound| > || -files | (/) | (/) | (/) | > || -archives | (/) | (/) | (/) | -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6836) exception thrown when accessing the job configuration web UI
Sangjin Lee created MAPREDUCE-6836: -- Summary: exception thrown when accessing the job configuration web UI Key: MAPREDUCE-6836 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6836 Project: Hadoop Map/Reduce Issue Type: Bug Components: webapps Affects Versions: 3.0.0-alpha2 Reporter: Sangjin Lee Priority: Minor When I navigate the MR job web UI and click the configuration link, the AM shows an exception: {noformat} 2017-01-25 11:40:55,521 ERROR [qtp2126664214-26] org.apache.hadoop.yarn.webapp.Dispatcher: error handling URI: /mapreduc e/conf/job_1485372765455_0002 java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:179) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1458) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:524) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.E
[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15817206#comment-15817206 ] Sangjin Lee commented on MAPREDUCE-5951: I went over the patch with Chris in some detail today, and am posting the review comments here for the record. (MRJobConfig.java) - {{mapreduce.job.jobjar.visibility}} and {{mapreduce.job.jobjar.sharedcache.uploadpolicy}} are computed values that are not user-facing; in that case we should not even define the defaults so that there is no confusion that these values are computed (JobResourceUploader.java) - l.159-160: I understand lines 161-165 are there to support programmatic use cases of the distributed cache that come in outside of the job submitter code path. Can we make the comments clearer so that the intent of this comes through? We could also annotate them in {{MRJobConfig}}. - l.171-172: it might be slightly better to use {{LinkedHashMap}}. That way, we'd have a predictable iteration order (the order in which they are specified in the user values). - l.219: To be fair, this is a bug we need to fix, right? Then can we file a JIRA and add the JIRA id here? - l.237-240: I would not worry about handling previous values here. Having duplicate paths is not really supported and the worst case scenario here is to reset this upload policy with the same value. - l.260-263: I think we can improve on this, and reconcile the shared cache with the wildcard feature. We could see if any resource is uploaded to the staging directory, and if so, still preserve the wildcard entry. We also need to consider the case where the shared cache is disabled but the wildcard is enabled. - l.291-294: same comment as above - l.348-351: same comment as above - l.388: Since we're dealing with a local filesystem URI based on l.381-382, the authority check is not meaningful. We should remove this check. - On a larger note, the path/URI handling between the job jar, libjars, files, and archives is not very consistent, which is an existing behavior. We need to see if they need to get the same consistent treatment for this to work. (Job.java) - l.1446: I realized later that passing an empty map has an effect of nulling out the config value; perhaps we could make that more explicit in the javadoc and/or comments/code? - l.1449-1463: nit: it might be slightly easier to read by using a simple string concatenation with "+" (JVM internally uses the {{StringBuilder}}) - l.1490: here also it might be better to use {{LinkedHashMap}} > Add support for the YARN Shared Cache > - > > Key: MAPREDUCE-5951 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5951-trunk-v1.patch, > MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, > MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, > MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, > MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, > MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, > MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, > MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch, > MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, > MAPREDUCE-5951-trunk.018.patch > > > Implement the necessary changes so that the MapReduce application can > leverage the new YARN shared cache (i.e. YARN-1492). > Specifically, allow per-job configuration so that MapReduce jobs can specify > which set of resources they would like to cache (i.e. jobjar, libjars, > archives, files). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl
[ https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6818: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: YARN-5355 Status: Resolved (was: Patch Available) Committed the patch to YARN-5355 and YARN-5355-branch-2. Thanks [~gtCarrera9] for your contribution! > Remove direct reference to TimelineClientImpl > - > > Key: MAPREDUCE-6818 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu > Labels: newbie++ > Fix For: YARN-5355 > > Attachments: YARN-5974-YARN-5355.001.patch > > > [~sjlee0]'s quick audit shows that things that are referencing > TimelineClientImpl directly today: > JobHistoryFileReplayMapperV1 (MR) > SimpleEntityWriterV1 (MR) > TestDistributedShell (DS) > TestDSAppMaster (DS) > TestNMTimelinePublisher (node manager) > TestTimelineWebServicesWithSSL (AHS) > This is not the right way to use TimelineClient and we should avoid direct > reference to TimelineClientImpl as much as possible. > Any newcomers to the community are more than welcome to take this. If this > remains unassigned for ~24hrs I'll jump in and do a quick fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl
[ https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6818: --- Target Version/s: YARN-5355 > Remove direct reference to TimelineClientImpl > - > > Key: MAPREDUCE-6818 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu > Labels: newbie++ > Attachments: YARN-5974-YARN-5355.001.patch > > > [~sjlee0]'s quick audit shows that things that are referencing > TimelineClientImpl directly today: > JobHistoryFileReplayMapperV1 (MR) > SimpleEntityWriterV1 (MR) > TestDistributedShell (DS) > TestDSAppMaster (DS) > TestNMTimelinePublisher (node manager) > TestTimelineWebServicesWithSSL (AHS) > This is not the right way to use TimelineClient and we should avoid direct > reference to TimelineClientImpl as much as possible. > Any newcomers to the community are more than welcome to take this. If this > remains unassigned for ~24hrs I'll jump in and do a quick fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl
[ https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6818: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-6732 > Remove direct reference to TimelineClientImpl > - > > Key: MAPREDUCE-6818 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu > Labels: newbie++ > Attachments: YARN-5974-YARN-5355.001.patch > > > [~sjlee0]'s quick audit shows that things that are referencing > TimelineClientImpl directly today: > JobHistoryFileReplayMapperV1 (MR) > SimpleEntityWriterV1 (MR) > TestDistributedShell (DS) > TestDSAppMaster (DS) > TestNMTimelinePublisher (node manager) > TestTimelineWebServicesWithSSL (AHS) > This is not the right way to use TimelineClient and we should avoid direct > reference to TimelineClientImpl as much as possible. > Any newcomers to the community are more than welcome to take this. If this > remains unassigned for ~24hrs I'll jump in and do a quick fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl
[ https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee moved YARN-5974 to MAPREDUCE-6818: -- Affects Version/s: (was: YARN-5355) Component/s: (was: timelineserver) Key: MAPREDUCE-6818 (was: YARN-5974) Project: Hadoop Map/Reduce (was: Hadoop YARN) > Remove direct reference to TimelineClientImpl > - > > Key: MAPREDUCE-6818 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Li Lu >Assignee: Li Lu > Labels: newbie++ > Attachments: YARN-5974-YARN-5355.001.patch > > > [~sjlee0]'s quick audit shows that things that are referencing > TimelineClientImpl directly today: > JobHistoryFileReplayMapperV1 (MR) > SimpleEntityWriterV1 (MR) > TestDistributedShell (DS) > TestDSAppMaster (DS) > TestNMTimelinePublisher (node manager) > TestTimelineWebServicesWithSSL (AHS) > This is not the right way to use TimelineClient and we should avoid direct > reference to TimelineClientImpl as much as possible. > Any newcomers to the community are more than welcome to take this. If this > remains unassigned for ~24hrs I'll jump in and do a quick fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626604#comment-15626604 ] Sangjin Lee commented on MAPREDUCE-6704: I am +1 for #3. There is the yuck factor of mixing up MR into YARN, but the issue is significant enough and the change small enough that I think we can make an exception here. We can move the discussion to the mailing list to expand the audience. > Container fail to launch for mapred application > --- > > Key: MAPREDUCE-6704 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Bibin A Chundatt >Priority: Blocker > Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch, > container-whitelist-env-wip.patch > > > Container fail to launch for mapred application. > As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set > .After > https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c >{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}} > since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false. > {noformat} > 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with > state FAILED due to: Application application_1462155939310_0004 failed 2 > times due to AM Container for appattempt_1462155939310_0004_02 exited > with exitCode: 1 > Failing this attempt.Diagnostics: Exception from container-launch. > Container id: container_1462155939310_0004_02_01 > Exit code: 1 > Stack trace: ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) > at org.apache.hadoop.util.Shell.run(Shell.java:850) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 1. Last 4096 bytes of stderr : > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; > support was removed in 8.0 > Error: Could not find or load main class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster > Container exited with a non-zero exit code 1. Last 4096 bytes of stderr : > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; > support was removed in 8.0 > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6741) add MR support to redact job conf properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570269#comment-15570269 ] Sangjin Lee commented on MAPREDUCE-6741: According to [the discussion|https://s.apache.org/Qu1g] we were having on the release cadence and EOL, the 2.6.x line could be EOLed some time in Nov. 2016. Although it is far from being concluded, the fate of another 2.6 release is at least uncertain. Why don't we hold off on backporting it to 2.6? We should probably raise the release cadence and EOL topic again and conclude it at some point... > add MR support to redact job conf properties > > > Key: MAPREDUCE-6741 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: mapreduce6741.001.patch, mapreduce6741.002.patch, > mapreduce6741.003.patch, mapreduce6741.004.patch, mapreduce6741.005.patch, > mapreduce6741.006.patch, mapreduce6741.branch-2.8.patch > > > JHS today displays all Job conf properties in Web UI directly. Users may have > some credentials or any sensitive information they added to the job conf but > do not want to be shown in Web UI. It'd be nice if we can allow users to > specify a set of properties which JHS will filter out when Job conf is > displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6779) Mapreduce job failure on submission
[ https://issues.apache.org/jira/browse/MAPREDUCE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee resolved MAPREDUCE-6779. Resolution: Resolved Assignee: Sangjin Lee > Mapreduce job failure on submission > --- > > Key: MAPREDUCE-6779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6779 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Sangjin Lee >Priority: Blocker > > Configure Hibench > Try running Enhanced TestDFSIO > {noformat} > 2016-09-15 18:20:24,849 INFO mapreduce.Job: Task Id : > attempt_1473943118844_0001_m_01_0, Status : FAILED > Error: java.lang.RuntimeException: Error in configuring object > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1806) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) > ... 9 more > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2330) > at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1108) > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) > ... 14 more > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > Class org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2298) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2322) > ... 16 more > Caused by: java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2202) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2296) > ... 17 more > {noformat} > *mapreduce.JobResourceUploader: No job jar file set. User classes may not be > found. See Job or Job#setJar(String).* > Job jar is not getting set and not uploaded to staging dir -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6779) Mapreduce job failure on submission
[ https://issues.apache.org/jira/browse/MAPREDUCE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15493447#comment-15493447 ] Sangjin Lee commented on MAPREDUCE-6779: Can you provide the hadoop jar command line with which you see this issue? Does it happen with any job submission or only with this job? > Mapreduce job failure on submission > --- > > Key: MAPREDUCE-6779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6779 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Bibin A Chundatt >Priority: Blocker > > Configure Hibench > Try running Enhanced TestDFSIO > {noformat} > 2016-09-15 18:20:24,849 INFO mapreduce.Job: Task Id : > attempt_1473943118844_0001_m_01_0, Status : FAILED > Error: java.lang.RuntimeException: Error in configuring object > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1806) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) > ... 9 more > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2330) > at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1108) > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) > ... 14 more > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > Class org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2298) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2322) > ... 16 more > Caused by: java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2202) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2296) > ... 17 more > {noformat} > *mapreduce.JobResourceUploader: No job jar file set. User classes may not be > found. See Job or Job#setJar(String).* > Job jar is not getting set and not uploaded to staging dir -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6637) Testcase Failure : TestFileInputFormat.testSplitLocationInfo
[ https://issues.apache.org/jira/browse/MAPREDUCE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6637: --- Fix Version/s: 2.6.5 Cherry-picked it to 2.6.5 (trivial). > Testcase Failure : TestFileInputFormat.testSplitLocationInfo > > > Key: MAPREDUCE-6637 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6637 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Fix For: 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6637.patch > > > Following testcase is failing after HADOOP-12810 > {noformat} > FAILED: org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo[0] > Error Message: > expected:<2> but was:<1> > Stack Trace: > java.lang.AssertionError: expected:<2> but was:<1> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo(TestFileInputFormat.java:115) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5817) Mappers get rescheduled on node transition even after all reducers are completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-5817: --- Fix Version/s: 2.6.5 Cherry-picked it into 2.6.5. > Mappers get rescheduled on node transition even after all reducers are > completed > > > Key: MAPREDUCE-5817 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5817 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.3.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: MAPREDUCE-5817.001.patch, MAPREDUCE-5817.002.patch, > mapreduce-5817.patch > > > We're seeing a behavior where a job runs long after all reducers were already > finished. We found that the job was rescheduling and running a number of > mappers beyond the point of reducer completion. In one situation, the job ran > for some 9 more hours after all reducers completed! > This happens because whenever a node transition (to an unusable state) comes > into the app master, it just reschedules all mappers that already ran on the > node in all cases. > Therefore, if any node transition has a potential to extend the job period. > Once this window opens, another node transition can prolong it, and this can > happen indefinitely in theory. > If there is some instability in the pool (unhealthy, etc.) for a duration, > then any big job is severely vulnerable to this problem. > If all reducers have been completed, JobImpl.actOnUnusableNode() should not > reschedule mapper tasks. If all reducers are completed, the mapper outputs > are no longer needed, and there is no need to reschedule mapper tasks as they > would not be consumed anyway. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6365) Refactor JobResourceUploader#uploadFilesInternal
[ https://issues.apache.org/jira/browse/MAPREDUCE-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6365: --- Fix Version/s: (was: 3.0.0-alpha2) 2.9.0 > Refactor JobResourceUploader#uploadFilesInternal > > > Key: MAPREDUCE-6365 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6365 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6365-trunk-v1.patch > > > JobResourceUploader#uploadFilesInternal is a large method and there are > similar pieces of code that could probably be pulled out into separate > methods. This refactor would improve readability of the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6365) Refactor JobResourceUploader#uploadFilesInternal
[ https://issues.apache.org/jira/browse/MAPREDUCE-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6365: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 Status: Resolved (was: Patch Available) Committed the patch to trunk. Thanks [~ctrezzo] for your contribution! Should this be backported to an earlier version? Branch-2 (2.9.0)? > Refactor JobResourceUploader#uploadFilesInternal > > > Key: MAPREDUCE-6365 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6365 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-6365-trunk-v1.patch > > > JobResourceUploader#uploadFilesInternal is a large method and there are > similar pieces of code that could probably be pulled out into separate > methods. This refactor would improve readability of the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6365) Refactor JobResourceUploader#uploadFilesInternal
[ https://issues.apache.org/jira/browse/MAPREDUCE-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385168#comment-15385168 ] Sangjin Lee commented on MAPREDUCE-6365: +1. Will commit soon. > Refactor JobResourceUploader#uploadFilesInternal > > > Key: MAPREDUCE-6365 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6365 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Attachments: MAPREDUCE-6365-trunk-v1.patch > > > JobResourceUploader#uploadFilesInternal is a large method and there are > similar pieces of code that could probably be pulled out into separate > methods. This refactor would improve readability of the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6735) Performance degradation caused by MAPREDUCE-5465 and HADOOP-12107
[ https://issues.apache.org/jira/browse/MAPREDUCE-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384866#comment-15384866 ] Sangjin Lee commented on MAPREDUCE-6735: It is certainly surprising that HADOOP-12107 is making a difference on terasort. What version of hadoop are you using for your test? Java version? Is it repeatable (i.e. the gap shows up consistently)? FYI, the nature of HADOOP-12107 has to do with *when* to clean up a certain data ( {{allData}} ) inside the {{FileSystem.Statistics}} objects. Before this change, it would get cleaned up when the owner thread gets garbage collected *and* a read operation is done on the {{Statistics}} object. By read operations I mean methods such as {{getBytesRead()}} and so on. After this change, the timing of this clean-up no longer depends on the read operations, and it will be done promptly when the thread is garbage collected. So in a sense, the change first ensures there is clean-up no matter what, and also moves up the timing of the clean-up. The worst-case scenario in which this can have a negative impact on performance is if the use case *never* reads the statistics. Prior to the change, as long as the heap can contain these objects, no clean-up will be done. With the change now we do perform additional clean-up on threads garbage collection. A subsequent observation is that the impact of the clean-up is greater if there is a *high degree of thread churn* within the JVM. If we're talking about only a handful of threads or long-lived threads, there should really be no difference. I would greatly appreciate it if you could dig a little deeper via logging or low overhead profiling to pinpoint the correlation. Thanks. > Performance degradation caused by MAPREDUCE-5465 and HADOOP-12107 > - > > Key: MAPREDUCE-6735 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6735 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Alexandr Balitsky > > Two commits, MAPREDUCE-5465 and HADOOP-12107 are making Terasort on YARN 10% > slower. > Reduce phase with those commits ~5 mins > Reduce phase without ~3.5 mins > Average Reduce is taking 4mins, 16sec with those commits compared to 3mins, > 48sec without. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6731: --- Attachment: MAPREDUCE-6731.02.patch Posted patch v.2. A small change in the directory name (to make it shorter). > TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail > for concurrent tests > - > > Key: MAPREDUCE-6731 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: MAPREDUCE-6731.01.patch, MAPREDUCE-6731.02.patch > > > {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses > the default file-system storage directory, and is brittle against concurrent > tests. > We should use a unique storage directory for the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371681#comment-15371681 ] Sangjin Lee commented on MAPREDUCE-6731: The test failure is unrelated. I would greatly appreciate your review. Thanks! > TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail > for concurrent tests > - > > Key: MAPREDUCE-6731 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: MAPREDUCE-6731.01.patch > > > {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses > the default file-system storage directory, and is brittle against concurrent > tests. > We should use a unique storage directory for the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371426#comment-15371426 ] Sangjin Lee commented on MAPREDUCE-6732: I have just created and pushed branches {{YARN-5355}} and {{YARN-5355-branch-2}}. They are now open for this feature development. > mapreduce tasks for YARN Timeline Service v.2: alpha 2 > -- > > Key: MAPREDUCE-6732 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline > Service v.2 alpha 2. > This is developed on feature branches: {{YARN-5355}} for the trunk-based > development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any > subtask work on this JIRA will be committed to those 2 branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6732: --- Description: This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline Service v.2 alpha 2. This is developed on feature branches: {{YARN-5355}} for the trunk-based development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any subtask work on this JIRA will be committed to those 2 branches. was:This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline Service v.2. > mapreduce tasks for YARN Timeline Service v.2: alpha 2 > -- > > Key: MAPREDUCE-6732 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline > Service v.2 alpha 2. > This is developed on feature branches: {{YARN-5355}} for the trunk-based > development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any > subtask work on this JIRA will be committed to those 2 branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6731: --- Status: Patch Available (was: In Progress) > TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail > for concurrent tests > - > > Key: MAPREDUCE-6731 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: MAPREDUCE-6731.01.patch > > > {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses > the default file-system storage directory, and is brittle against concurrent > tests. > We should use a unique storage directory for the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6731: --- Attachment: MAPREDUCE-6731.01.patch Posted patch v.1. This uses a test directory local to the build, instead of the global shared storage directory. I also made the path separator cross-platform. > TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail > for concurrent tests > - > > Key: MAPREDUCE-6731 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: MAPREDUCE-6731.01.patch > > > {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses > the default file-system storage directory, and is brittle against concurrent > tests. > We should use a unique storage directory for the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6731 started by Sangjin Lee. -- > TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail > for concurrent tests > - > > Key: MAPREDUCE-6731 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses > the default file-system storage directory, and is brittle against concurrent > tests. > We should use a unique storage directory for the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6318) Refactor JobHistoryEventHandler for handling events in ATS v1 after YARN-2928 merge back to trunk and branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6318: --- Parent Issue: MAPREDUCE-6732 (was: MAPREDUCE-6331) > Refactor JobHistoryEventHandler for handling events in ATS v1 after YARN-2928 > merge back to trunk and branch-2 > --- > > Key: MAPREDUCE-6318 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6318 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobhistoryserver >Reporter: Junping Du >Assignee: Junping Du > > Per discussion in YARN-3046, we need to refactor the process flow for > timeline events (ATS v1) in JobHistoryEventHandler. The refactor work should > get happen after YARN-2928 merged into trunk and branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2
Sangjin Lee created MAPREDUCE-6732: -- Summary: mapreduce tasks for YARN Timeline Service v.2: alpha 2 Key: MAPREDUCE-6732 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Sangjin Lee Assignee: Sangjin Lee This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline Service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests
Sangjin Lee created MAPREDUCE-6731: -- Summary: TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests Key: MAPREDUCE-6731 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 3.0.0-alpha1 Reporter: Sangjin Lee Assignee: Sangjin Lee {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses the default file-system storage directory, and is brittle against concurrent tests. We should use a unique storage directory for the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6331) [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)
[ https://issues.apache.org/jira/browse/MAPREDUCE-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee resolved MAPREDUCE-6331. Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha1 It's been merged to trunk. Huge thanks to the contributors who worked on this feature ([~jrottinghuis], [~djp], [~gtCarrera9], [~Naganarasimha], [~varun_saxena], [~vinodkv], [~vrushalic], and [~zjshen]) and everyone who participated in reviews and feedback! > [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928) > > > Key: MAPREDUCE-6331 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6331 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Sangjin Lee > Fix For: 3.0.0-alpha1 > > > Tracking umbrella for all MR changes to make it work with Timeline Service > Nextgen - YARN-2928. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343119#comment-15343119 ] Sangjin Lee commented on MAPREDUCE-6720: Please feel free to go ahead and commit it. Thanks! > Inconsistent values of counters across tasks and job reported to timeline > service. > -- > > Key: MAPREDUCE-6720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: Counters For Job.png, Counters for Tasks.png, > MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch, > MAPREDUCE-6720-YARN-2928.03.patch, MAPREDUCE-6720-YARN-2928.04.patch > > > While testing found below issue. For some of the task counters, we do not > have consistent values. This is not the case with every counter though. > Consider the case of counter > "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN". > I found that its value for a flow I ran, was 936018 bytes. For the 3 apps > associated with this flow run, the values were 312006 bytes each (which > equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further > down I found though that for one of the apps, the 4 tasks(2 mappers and 2 > reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes > each for the 2 mappers. > This means the value reported for the app should be (2 * 156003 + 2* 155918) > or 623842 bytes but it is only 312006 bytes which indicates that only counter > value of mappers is being picked up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342657#comment-15342657 ] Sangjin Lee commented on MAPREDUCE-6720: The latest patch looks good to me. Please let us know if you're OK with the approach and the patch. Thanks! > Inconsistent values of counters across tasks and job reported to timeline > service. > -- > > Key: MAPREDUCE-6720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: Counters For Job.png, Counters for Tasks.png, > MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch, > MAPREDUCE-6720-YARN-2928.03.patch, MAPREDUCE-6720-YARN-2928.04.patch > > > While testing found below issue. For some of the task counters, we do not > have consistent values. This is not the case with every counter though. > Consider the case of counter > "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN". > I found that its value for a flow I ran, was 936018 bytes. For the 3 apps > associated with this flow run, the values were 312006 bytes each (which > equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further > down I found though that for one of the apps, the 4 tasks(2 mappers and 2 > reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes > each for the 2 mappers. > This means the value reported for the app should be (2 * 156003 + 2* 155918) > or 623842 bytes but it is only 312006 bytes which indicates that only counter > value of mappers is being picked up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342402#comment-15342402 ] Sangjin Lee commented on MAPREDUCE-6720: The latest patch looks good to me, pending the checkstyle and whitespace fixes and the jenkins run. Again, let's hold off until the rebase is complete before committing this. > Inconsistent values of counters across tasks and job reported to timeline > service. > -- > > Key: MAPREDUCE-6720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: Counters For Job.png, Counters for Tasks.png, > MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch, > MAPREDUCE-6720-YARN-2928.03.patch > > > While testing found below issue. For some of the task counters, we do not > have consistent values. This is not the case with every counter though. > Consider the case of counter > "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN". > I found that its value for a flow I ran, was 936018 bytes. For the 3 apps > associated with this flow run, the values were 312006 bytes each (which > equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further > down I found though that for one of the apps, the 4 tasks(2 mappers and 2 > reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes > each for the 2 mappers. > This means the value reported for the app should be (2 * 156003 + 2* 155918) > or 623842 bytes but it is only 312006 bytes which indicates that only counter > value of mappers is being picked up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6719) The list of -libjars archives should be replaced with a wildcard in the distributed cache to reduce the application footprint in the state store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6719: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.9.0 Status: Resolved (was: Patch Available) Committed patch v.2 to trunk and branch-2 (2.9.0). Thanks [~templedf] for your contribution! > The list of -libjars archives should be replaced with a wildcard in the > distributed cache to reduce the application footprint in the state store > > > Key: MAPREDUCE-6719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distributed-cache >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6719.001.patch, MAPREDUCE-6719.002.patch > > > When using the -libjars option to add classes to the classpath, every library > so added is explicitly listed in the ContainerLaunchContext's local resources > even though they're all uploaded to the same directory in HDFS. When using > tools like Crunch without an uber JAR or when trying to take advantage of the > shared cache, the number of libraries can be quite large. We've seen many > cases where we had to turn down the max number of applications to prevent ZK > from running out of heap because of the size of the state store entries. > This JIRA proposes to allow for wildcards both in the internal processing of > the -libjars switch and in paths added through the Job and DistributedCache > classes. Rather than listing all files independently, this JIRA proposes to > replace the complete list of libdir files with the wildcarded libdir > directory, e.g. "libdir/*". This behavior is the same as the current behavior > when using -libjars, but avoids explicitly listing every file. > This capability will also be exposed by the > {{DistributedCache.addCacheFile()}} method. > See YARN-4958 for the NM side of the implementation and additional discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6719) The list of -libjars archives should be replaced with a wildcard in the distributed cache to reduce the application footprint in the state store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342378#comment-15342378 ] Sangjin Lee commented on MAPREDUCE-6719: Thanks Daniel! Will commit it shortly. > The list of -libjars archives should be replaced with a wildcard in the > distributed cache to reduce the application footprint in the state store > > > Key: MAPREDUCE-6719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distributed-cache >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: MAPREDUCE-6719.001.patch, MAPREDUCE-6719.002.patch > > > When using the -libjars option to add classes to the classpath, every library > so added is explicitly listed in the ContainerLaunchContext's local resources > even though they're all uploaded to the same directory in HDFS. When using > tools like Crunch without an uber JAR or when trying to take advantage of the > shared cache, the number of libraries can be quite large. We've seen many > cases where we had to turn down the max number of applications to prevent ZK > from running out of heap because of the size of the state store entries. > This JIRA proposes to allow for wildcards both in the internal processing of > the -libjars switch and in paths added through the Job and DistributedCache > classes. Rather than listing all files independently, this JIRA proposes to > replace the complete list of libdir files with the wildcarded libdir > directory, e.g. "libdir/*". This behavior is the same as the current behavior > when using -libjars, but avoids explicitly listing every file. > This capability will also be exposed by the > {{DistributedCache.addCacheFile()}} method. > See YARN-4958 for the NM side of the implementation and additional discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342368#comment-15342368 ] Sangjin Lee commented on MAPREDUCE-6720: [~varun_saxena], could you look at the checkstyle and whitespace issues? Thanks! > Inconsistent values of counters across tasks and job reported to timeline > service. > -- > > Key: MAPREDUCE-6720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: Counters For Job.png, Counters for Tasks.png, > MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch, > MAPREDUCE-6720-YARN-2928.03.patch > > > While testing found below issue. For some of the task counters, we do not > have consistent values. This is not the case with every counter though. > Consider the case of counter > "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN". > I found that its value for a flow I ran, was 936018 bytes. For the 3 apps > associated with this flow run, the values were 312006 bytes each (which > equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further > down I found though that for one of the apps, the 4 tasks(2 mappers and 2 > reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes > each for the 2 mappers. > This means the value reported for the app should be (2 * 156003 + 2* 155918) > or 623842 bytes but it is only 312006 bytes which indicates that only counter > value of mappers is being picked up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342237#comment-15342237 ] Sangjin Lee commented on MAPREDUCE-6720: If we are going to store the map counters and reduce counters, we should make the metric names such that one can easily query for them only. So in that sense, making it a total prefix or suffix to the metric name might be a better idea? In other words, how about something like {noformat} org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for total. MAP:org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for MAP. REDUCE:org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for REDUCE. {noformat} or {noformat} org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for total. org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS:MAP for MAP. org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS:REDUCE for REDUCE. {noformat} ? > Inconsistent values of counters across tasks and job reported to timeline > service. > -- > > Key: MAPREDUCE-6720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: Counters For Job.png, Counters for Tasks.png, > MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch > > > While testing found below issue. For some of the task counters, we do not > have consistent values. This is not the case with every counter though. > Consider the case of counter > "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN". > I found that its value for a flow I ran, was 936018 bytes. For the 3 apps > associated with this flow run, the values were 312006 bytes each (which > equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further > down I found though that for one of the apps, the 4 tasks(2 mappers and 2 > reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes > each for the 2 mappers. > This means the value reported for the app should be (2 * 156003 + 2* 155918) > or 623842 bytes but it is only 312006 bytes which indicates that only counter > value of mappers is being picked up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342076#comment-15342076 ] Sangjin Lee commented on MAPREDUCE-6720: Thanks for reporting this issue [~varun_saxena]. It's a good catch. This reminded me of the same situation I encountered when I was looking at the job history file replaying in the performance test (https://github.com/apache/hadoop/blob/YARN-2928/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV2.java#L101). I am OK with the recording only the total counters. If we do want the mapper counters and reducer counters too, we could consider prefixing or suffixing the metric names to disambiguate them. > Inconsistent values of counters across tasks and job reported to timeline > service. > -- > > Key: MAPREDUCE-6720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: Counters For Job.png, Counters for Tasks.png, > MAPREDUCE-6720-YARN-2928.01.patch > > > While testing found below issue. For some of the task counters, we do not > have consistent values. This is not the case with every counter though. > Consider the case of counter > "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN". > I found that its value for a flow I ran, was 936018 bytes. For the 3 apps > associated with this flow run, the values were 312006 bytes each (which > equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further > down I found though that for one of the apps, the 4 tasks(2 mappers and 2 > reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes > each for the 2 mappers. > This means the value reported for the app should be (2 * 156003 + 2* 155918) > or 623842 bytes but it is only 312006 bytes which indicates that only counter > value of mappers is being picked up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6719) -libjars should use wildcards to reduce the application footprint in the state store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340813#comment-15340813 ] Sangjin Lee edited comment on MAPREDUCE-6719 at 6/21/16 1:03 AM: - The latest patch looks good to me. The unit test that timed out is known to happen occasionally (can't locate an open JIRA for that one though). The deprecation warning is also an unfortunate result of {{DistributedCache}} being deprecated. I do have a small quibble with the title of the JIRA. The title may give you an impression that with this JIRA one could use a wildcard in specifying the libjars *argument*. But that's not the case. What this JIRA does (in conjunction with YARN-4958) is to collapse the libjar directory in the configuration as well as the container launch context. But the current title can be confused with HADOOP-12747 ("support wildcard in libjars argument"). Can we modify the title to better describe this JIRA? was (Author: sjlee0): The latest patch looks good to me. The unit test that timed out is known to happen occasionally (can't locate an open JIRA for that one though). The deprecation warning is also an unfortunately result of {{DistributedCache}} being deprecated. I do have a small quibble with the title of the JIRA. The title may give you an impression that with this JIRA one could use a wildcard in specifying the libjars *argument*. But that's not the case. What this JIRA does (in conjunction with YARN-4958) is to collapse the libjar directory in the configuration as well as the container launch context. But the current title can be confused with HADOOP-12747 ("support wildcard in libjars argument"). Can we modify the title to better describe this JIRA? > -libjars should use wildcards to reduce the application footprint in the > state store > > > Key: MAPREDUCE-6719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distributed-cache >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: MAPREDUCE-6719.001.patch, MAPREDUCE-6719.002.patch > > > When using the -libjars option to add classes to the classpath, every library > so added is explicitly listed in the ContainerLaunchContext's local resources > even though they're all uploaded to the same directory in HDFS. When using > tools like Crunch without an uber JAR or when trying to take advantage of the > shared cache, the number of libraries can be quite large. We've seen many > cases where we had to turn down the max number of applications to prevent ZK > from running out of heap because of the size of the state store entries. > This JIRA proposes to allow for wildcards both in the internal processing of > the -libjars switch and in paths added through the Job and DistributedCache > classes. Rather than listing all files independently, this JIRA proposes to > replace the complete list of libdir files with the wildcarded libdir > directory, e.g. "libdir/*". This behavior is the same as the current behavior > when using -libjars, but avoids explicitly listing every file. > This capability will also be exposed by the > {{DistributedCache.addCacheFile()}} method. > See YARN-4958 for the NM side of the implementation and additional discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6719) -libjars should use wildcards to reduce the application footprint in the state store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340813#comment-15340813 ] Sangjin Lee commented on MAPREDUCE-6719: The latest patch looks good to me. The unit test that timed out is known to happen occasionally (can't locate an open JIRA for that one though). The deprecation warning is also an unfortunately result of {{DistributedCache}} being deprecated. I do have a small quibble with the title of the JIRA. The title may give you an impression that with this JIRA one could use a wildcard in specifying the libjars *argument*. But that's not the case. What this JIRA does (in conjunction with YARN-4958) is to collapse the libjar directory in the configuration as well as the container launch context. But the current title can be confused with HADOOP-12747 ("support wildcard in libjars argument"). Can we modify the title to better describe this JIRA? > -libjars should use wildcards to reduce the application footprint in the > state store > > > Key: MAPREDUCE-6719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distributed-cache >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: MAPREDUCE-6719.001.patch, MAPREDUCE-6719.002.patch > > > When using the -libjars option to add classes to the classpath, every library > so added is explicitly listed in the ContainerLaunchContext's local resources > even though they're all uploaded to the same directory in HDFS. When using > tools like Crunch without an uber JAR or when trying to take advantage of the > shared cache, the number of libraries can be quite large. We've seen many > cases where we had to turn down the max number of applications to prevent ZK > from running out of heap because of the size of the state store entries. > This JIRA proposes to allow for wildcards both in the internal processing of > the -libjars switch and in paths added through the Job and DistributedCache > classes. Rather than listing all files independently, this JIRA proposes to > replace the complete list of libdir files with the wildcarded libdir > directory, e.g. "libdir/*". This behavior is the same as the current behavior > when using -libjars, but avoids explicitly listing every file. > This capability will also be exposed by the > {{DistributedCache.addCacheFile()}} method. > See YARN-4958 for the NM side of the implementation and additional discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6719) -libjars should use wildcards to reduce the application footprint in the state store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339939#comment-15339939 ] Sangjin Lee commented on MAPREDUCE-6719: {noformat} diff --git hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/nbproject/project.properties hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/nbproject/project.properties new file mode 100644 index 000..e69de29 {noformat} Is the above file supposed to be part of this patch? It seems to me that either the file's content should be in the patch or it should be removed from the patch. > -libjars should use wildcards to reduce the application footprint in the > state store > > > Key: MAPREDUCE-6719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distributed-cache >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: MAPREDUCE-6719.001.patch > > > When using the -libjars option to add classes to the classpath, every library > so added is explicitly listed in the ContainerLaunchContext's local resources > even though they're all uploaded to the same directory in HDFS. When using > tools like Crunch without an uber JAR or when trying to take advantage of the > shared cache, the number of libraries can be quite large. We've seen many > cases where we had to turn down the max number of applications to prevent ZK > from running out of heap because of the size of the state store entries. > This JIRA proposes to allow for wildcards both in the internal processing of > the -libjars switch and in paths added through the Job and DistributedCache > classes. Rather than listing all files independently, this JIRA proposes to > replace the complete list of libdir files with the wildcarded libdir > directory, e.g. "libdir/*". This behavior is the same as the current behavior > when using -libjars, but avoids explicitly listing every file. > This capability will also be exposed by the > {{DistributedCache.addCacheFile()}} method. > See YARN-4958 for the NM side of the implementation and additional discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6688) Store job configurations in Timeline Service v2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6688: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: YARN-2928 Status: Resolved (was: Patch Available) Committed the patch (04). Thanks [~varun_saxena] for your contribution, and [~gtCarrera9], [~Naganarasimha], and [~vrushalic] for your reviews! > Store job configurations in Timeline Service v2 > --- > > Key: MAPREDUCE-6688 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: MAPREDUCE-6688-YARN-2928.01.patch, > MAPREDUCE-6688-YARN-2928.02.patch, MAPREDUCE-6688-YARN-2928.03.patch, > MAPREDUCE-6688-YARN-2928.04.patch, MAPREDUCE-6688-YARN-2928.v2.01.patch, > MAPREDUCE-6688-YARN-2928.v2.02.patch, YARN-3959-YARN-2928.01.patch > > > We already have configuration field in HBase schema for application entity. > We need to make sure AM write it out when it get launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6688) Store job configurations in Timeline Service v2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6688: --- Attachment: MAPREDUCE-6688-YARN-2928.04.patch Removed several unused imports for the checkstyle issues. > Store job configurations in Timeline Service v2 > --- > > Key: MAPREDUCE-6688 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6688-YARN-2928.01.patch, > MAPREDUCE-6688-YARN-2928.02.patch, MAPREDUCE-6688-YARN-2928.03.patch, > MAPREDUCE-6688-YARN-2928.04.patch, MAPREDUCE-6688-YARN-2928.v2.01.patch, > MAPREDUCE-6688-YARN-2928.v2.02.patch, YARN-3959-YARN-2928.01.patch > > > We already have configuration field in HBase schema for application entity. > We need to make sure AM write it out when it get launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268123#comment-15268123 ] Sangjin Lee commented on MAPREDUCE-6688: The latest patch (v2.02) looks good. I'll wait for jenkins to come back. > Store job configurations in Timeline Service v2 > --- > > Key: MAPREDUCE-6688 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6688-YARN-2928.01.patch, > MAPREDUCE-6688-YARN-2928.02.patch, MAPREDUCE-6688-YARN-2928.03.patch, > MAPREDUCE-6688-YARN-2928.v2.01.patch, MAPREDUCE-6688-YARN-2928.v2.02.patch, > YARN-3959-YARN-2928.01.patch > > > We already have configuration field in HBase schema for application entity. > We need to make sure AM write it out when it get launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15267638#comment-15267638 ] Sangjin Lee commented on MAPREDUCE-6688: Agreeing with [~gtCarrera9], I'd like to propose that we proceed with the version without refactoring for now. What do others think? I took a look at that patch (v.2.01), and had a few comments. (JobHistoryEventHandler.java) - l.1129-1130: The id and type are set twice (once in {{createJobEntity()}}, another here). I think these second set calls should be removed. - l.1140-1145: if a single key/value exceeds the limit, it will do a spurious {{putEntities()}} call with empty entities unnecessarily; I think there should be an additional check to see if there is anything to send Did we reach a conclusion on whether these should be {{putEntities()}} (blocking) or {{putEntitiesAsync()}}? I'm leaning slightly towards the blocking call, as the config is arguably more important information than periodic metric updates for example. I don't think we have established a firm rule on what should be sync and async, however, and we might need to revisit it as a whole later. > Store job configurations in Timeline Service v2 > --- > > Key: MAPREDUCE-6688 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6688-YARN-2928.01.patch, > MAPREDUCE-6688-YARN-2928.02.patch, MAPREDUCE-6688-YARN-2928.03.patch, > MAPREDUCE-6688-YARN-2928.v2.01.patch, YARN-3959-YARN-2928.01.patch > > > We already have configuration field in HBase schema for application entity. > We need to make sure AM write it out when it get launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265642#comment-15265642 ] Sangjin Lee commented on MAPREDUCE-6688: If we don't have enough time to resolve these points, I think we have options to do the simple thing (emit the config in a single call and defer the size consideration in a later JIRA) or even defer this JIRA itself. > Store job configurations in Timeline Service v2 > --- > > Key: MAPREDUCE-6688 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6688-YARN-2928.01.patch, > YARN-3959-YARN-2928.01.patch > > > We already have configuration field in HBase schema for application entity. > We need to make sure AM write it out when it get launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265640#comment-15265640 ] Sangjin Lee commented on MAPREDUCE-6424: Thanks [~Naganarasimha]! I'm +1 on the latest patch. I'd like to give a little time for [~gtCarrera9] and [~varun_saxena]. Could you guys review it, and commit it if you are +1 too? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch, MAPREDUCE-6424-YARN-2928.v1.005.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265362#comment-15265362 ] Sangjin Lee commented on MAPREDUCE-6424: Thanks for tracking down the unit test failure until late/early. The checkstyle issues look trivial to correct. [~Naganarasimha], could I bother you to fix them? Then I'd be +1 on the patch. What do other think? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265194#comment-15265194 ] Sangjin Lee commented on MAPREDUCE-6688: I am comfortable with a constant (not much users would gain by tweaking this value high or low), but I'd like to hear what others think... > Store job configurations in Timeline Service v2 > --- > > Key: MAPREDUCE-6688 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-3959-YARN-2928.01.patch > > > We already have configuration field in HBase schema for application entity. > We need to make sure AM write it out when it get launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264842#comment-15264842 ] Sangjin Lee commented on MAPREDUCE-6688: bq. If we have done something in hRaven, we can probably adopt the same approach if conflicting opinions do not exist. cc [~vrushalic] and [~jrottinghuis] bq. Maybe we can go ahead with splitting with a fixed static constant for size of configs to be sent in one shot ? Is 100 kb fine ? Thoughts ? I'm +1 on doing multiple {{putEntities()}} calls with the upper bound of 100 KB with one exception; if a single value exceeds that size. We just need to decide on a behavior, and I would lean towards still sending it for now if a single key/value exceeds the upper bound. Would it be feasible? > Store job configurations in Timeline Service v2 > --- > > Key: MAPREDUCE-6688 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-3959-YARN-2928.01.patch > > > We already have configuration field in HBase schema for application entity. > We need to make sure AM write it out when it get launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264714#comment-15264714 ] Sangjin Lee commented on MAPREDUCE-6424: I'm fine with the suggestion. Be reminded that the timeline client combines multiple (async) writes to arrive at a reasonable level of writes to the timeline collector. So in general, the penalty for making fine-grained putEntitiesAsync() calls shouldn't be too big. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264686#comment-15264686 ] Sangjin Lee commented on MAPREDUCE-6424: Yeah that might be a better fit. In general, any place where we have a need to loop to wait until a test condition is satisfied is a candidate to consider {{GenericTestUtils.waitFor()}}. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263116#comment-15263116 ] Sangjin Lee commented on MAPREDUCE-6424: (JobHistoryEventHandler.java) - l.1053: I believe this causes a checkstyle violation (line too long) - l.1054: {{timestamp}} here is no longer used, and should be removed (JobFinishedEvent.java) - l.161: Sorry I should have been clearer, but since we have a clear timestamp associated with this event ({{finishTime}}), I think we ought to use that time over the current time millis. Thoughts? Same for {{MapAttemptFinishedEvent}}, {{ReduceAttemptFinishedEvent}}, {{TaskAttemptFinishedEvent}}, {{TaskAttemptUnsuccessfulCompletionEvent}}, {{TaskFailedEvent}}, and {{TaskFinishedEvent}} (TestMRTimelineEventHandling.java) - l.337-359: Is it possible to use {{GenericTestUtils.waitFor()}} for this? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262545#comment-15262545 ] Sangjin Lee commented on MAPREDUCE-6424: Please ignore my comment about the findbugs warnings. They appear unrelated to our branch. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262537#comment-15262537 ] Sangjin Lee commented on MAPREDUCE-6424: Thanks [~Naganarasimha] for the updated patch! It looks good for the most part. Some feedback: (JobHistoryEventHandler.java) - l.1055-1067: it might be better if we do this only if the metrics is not null (i.e. moving the {{if (timelineMetrics != null)}}) check to the outside - l.1057, 1061: I don't think this event is needed at the application entity (all we need are the metrics) - l.1231: I think this is a debatable point, but since these are final metric values would it be better to write them via {{putEntities()}}? (JobHistoryEventUtils.java) - l.68: This might be a small point, but how about adding {{long timestamp}} as an input argument so that all metrics can have the same timestamp? That way we can precisely set the timestamp if needed. (TestMRTimelineEventHandling.java) - can we add a test to check the metrics for the application entity? It looks like we have a couple of pre-existing findbugs warnings. Did you have a chance to see if they are straightforward to fix? Also, the checkstyle violations with the patch should be easy to fix. Are the unit test failures unrelated? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259517#comment-15259517 ] Sangjin Lee commented on MAPREDUCE-6424: {quote} This seems to be a good idea for oozie like flows but just one query, would it be good for app to publish metrics in SystemEntities ? IIRC YARN-3401 was raised to block it later on ? so do we need to support some other mechanism to publish only metrics ? {quote} We may need to refine what we do on YARN-3401. YARN-3401 notwithstanding, we still have a need to support aggregating framework-owned metrics. Consider our famous example: HDFS bytes written for the job. If we emit it for the MR job entity, then it would not benefit from aggregation onto the flow and so on. One can query for the job entities and get the HDFS bytes written for individual jobs, but there would be no flow aggregation. By adding the metrics on the YARN application entity, the flow aggregation will kick in naturally and be able to show it at the flow run level. If we don't do this, then we need to find a way to transfer metrics from certain framework-specific entities to the application entities so that they can be aggregated. I would think that would be considerably more involved. {quote} Ok basically was trying to understand for the next jira, may be we can discuss more there. Query was : Apart from metrics getting published at the end of each task(M/R) we need to publish metrics when task(M/R) is running or have some way to share running task's (M/R) metrics to AM(via umbilical proto) and AM publish aggregated running tasks (M/R) metrics periodically ? {quote} I see. Yeah, I haven't given a lot of thoughts on exactly how we might do this. As you mentioned, we can carry on the discussion in a new JIRA. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259240#comment-15259240 ] Sangjin Lee commented on MAPREDUCE-6424: Thanks [~Naganarasimha] for your patch! I think we said it is OK to emit them as is and capture the work of emitting them periodically in a later JIRA. One thing to add, though: I mentioned it elsewhere but we also need to emit the metrics at the *YARN application* level if we want them aggregated to the flows. Therefore, we should add code that emits the metrics for the YARN application entity in addition to the MR job entity. I'd say that's actually even more important than metrics at the job entity. The metrics at the MR-specific entities are there to view, but they will not be aggregated. On the other hand, the YARN application entity is the focal point of real time flow aggregation. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6315) Implement retrieval of logs for crashed MR-AM via jhist in the staging directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258663#comment-15258663 ] Sangjin Lee commented on MAPREDUCE-6315: Thanks [~jira.shegalov]! This is a much-needed enhancement. I agree with most of Ming's comments. Especially it would be awesome if the web side can benefit from this. > Implement retrieval of logs for crashed MR-AM via jhist in the staging > directory > > > Key: MAPREDUCE-6315 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6315 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mr-am >Affects Versions: 2.7.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Critical > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-6315.001.patch, MAPREDUCE-6315.002.patch, > MAPREDUCE-6315.003.patch > > > When all AM attempts crash, there is no record of them in JHS. Thus no easy > way to get the logs. This JIRA automates the procedure by utilizing the jhist > file in the staging directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5817) Mappers get rescheduled on node transition even after all reducers are completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15246018#comment-15246018 ] Sangjin Lee commented on MAPREDUCE-5817: I have no objections to backporting this to branch-2.7 or branch-2.6. Thanks! > Mappers get rescheduled on node transition even after all reducers are > completed > > > Key: MAPREDUCE-5817 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5817 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.3.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.8.0 > > Attachments: MAPREDUCE-5817.001.patch, MAPREDUCE-5817.002.patch, > mapreduce-5817.patch > > > We're seeing a behavior where a job runs long after all reducers were already > finished. We found that the job was rescheduling and running a number of > mappers beyond the point of reducer completion. In one situation, the job ran > for some 9 more hours after all reducers completed! > This happens because whenever a node transition (to an unusable state) comes > into the app master, it just reschedules all mappers that already ran on the > node in all cases. > Therefore, if any node transition has a potential to extend the job period. > Once this window opens, another node transition can prolong it, and this can > happen indefinitely in theory. > If there is some instability in the pool (unhealthy, etc.) for a duration, > then any big job is severely vulnerable to this problem. > If all reducers have been completed, JobImpl.actOnUnusableNode() should not > reschedule mapper tasks. If all reducers are completed, the mapper outputs > are no longer needed, and there is no need to reschedule mapper tasks as they > would not be consumed anyway. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance
[ https://issues.apache.org/jira/browse/MAPREDUCE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee resolved MAPREDUCE-6372. Resolution: Resolved Fix Version/s: YARN-2928 I have addressed the known issues as part of MAPREDUCE-6546. > clean up several issues with TimelineServicePerformance > --- > > Key: MAPREDUCE-6372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > > We found a few issues with the TimelineServicePerformanceV2 test driver while > running it for the performance tests. Filing this JIRA to fix those issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187531#comment-15187531 ] Sangjin Lee commented on MAPREDUCE-6546: Thanks [~Naganarasimha] for your review and commit! > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: MAPREDUCE-6546-YARN-2928.01.patch, > MAPREDUCE-6546-YARN-2928.02.patch, MAPREDUCE-6546-YARN-2928.03.patch > > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184358#comment-15184358 ] Sangjin Lee commented on MAPREDUCE-6546: Oops sorry about that. I should have printed out the message before cutting the patch. Yes, please feel free to add the closing parenthesis. Thanks! > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6546-YARN-2928.01.patch, > MAPREDUCE-6546-YARN-2928.02.patch, MAPREDUCE-6546-YARN-2928.03.patch > > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6546: --- Attachment: MAPREDUCE-6546-YARN-2928.03.patch Posted patch v.3. Addressed Naga's feedback. Thanks for the review! > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6546-YARN-2928.01.patch, > MAPREDUCE-6546-YARN-2928.02.patch, MAPREDUCE-6546-YARN-2928.03.patch > > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6546: --- Attachment: MAPREDUCE-6546-YARN-2928.02.patch Posted patch v.2. > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6546-YARN-2928.01.patch, > MAPREDUCE-6546-YARN-2928.02.patch > > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157507#comment-15157507 ] Sangjin Lee commented on MAPREDUCE-6546: Thanks [~garlanaganarasi...@huawei.com] for your review. Those are good points. I'll post an updated patch shortly. > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6546-YARN-2928.01.patch > > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149805#comment-15149805 ] Sangjin Lee commented on MAPREDUCE-6546: The unit test issues are unrelated. > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6546-YARN-2928.01.patch > > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6546: --- Attachment: MAPREDUCE-6546-YARN-2928.01.patch Posted patch v.1. The test main class is now merged into {{TimelineServicePerformance}}. Also moved all v2-specific classes to the same package (o.a.h.mapreduce). The diff is somewhat hard to read because some classes and code were moved. Let me know if you want me to add more details to aid your review. > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6546-YARN-2928.01.patch > > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6546: --- Status: Patch Available (was: In Progress) > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6546-YARN-2928.01.patch > > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance
[ https://issues.apache.org/jira/browse/MAPREDUCE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6372 stopped by Sangjin Lee. -- > clean up several issues with TimelineServicePerformance > --- > > Key: MAPREDUCE-6372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > > We found a few issues with the TimelineServicePerformanceV2 test driver while > running it for the performance tests. Filing this JIRA to fix those issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6546: --- Labels: yarn-2928-1st-milestone (was: ) > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6546 started by Sangjin Lee. -- > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085803#comment-15085803 ] Sangjin Lee commented on MAPREDUCE-6577: Thanks for pointing that out. I somehow (mistakenly) thought that 2.8.0 was already being cut. I just picked the fix to 2.8.0. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4 > > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Fix Version/s: 2.8.0 > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4 > > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6588) default values for PATH for tasks are not quite correct on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6588: --- Summary: default values for PATH for tasks are not quite correct on Windows (was: default values for PATH for tasks are not quite correct) > default values for PATH for tasks are not quite correct on Windows > -- > > Key: MAPREDUCE-6588 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6588 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee > > There are several issues with the way PATH are handled for MR tasks and AMs > on Windows. > - it does not get the PWD as part of the path, unlike the non-Windows case > (see MAPREDUCE-6021) > - the MR AM doesn't get the Hadoop's bin directory in the path explicitly > through mapred configuration; rather, it implicitly inherits it from the node > manager (see MAPREDUCE-6577) > - the default value for the MR task PATH uses "%...%" macros which may be > expanded on the client side prematurely; this is probably OK for tasks (as > it's done by the AM), but not OK for AMs; the late-binding cross-platform > macros should be used > I took a stab at addressing these as part of MAPREDUCE-6577, but I realized > that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus, > I'm filing a separate JIRA to address the Windows side of things. > On a related note, there are quite a few brittle tests on testing these > (TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such > a way that the tests pass against the current behavior but don't provide a > lot of robust test value. We should also update those tests to make them more > robust. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.6.4 2.9.0 2.7.3 Status: Resolved (was: Patch Available) Committed the patch to trunk, branch-2, branch-2.7, and branch-2.6. Thanks [~jlowe] and [~cnauroth] for your reviews and suggestions! > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Fix For: 2.7.3, 2.9.0, 2.6.4 > > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083856#comment-15083856 ] Sangjin Lee commented on MAPREDUCE-6577: Thanks [~cnauroth]! [~jlowe], do let me know if you're +1 on the latest patch. I can commit it then. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Attachment: MAPREDUCE-6577.04.patch Posted patch v.4. This keeps the Windows behavior identical to the current trunk. As a result, some of the code changes are slightly uglier. :) Another interesting change: as part of testing this pretty extensively, I stumbled onto the fact that HADOOP_COMMON_HOME was not being set in the hadoop unit tests. There are a lot of places where HADOOP_COMMON_HOME is used (including this one), and I suppose this problem was masked because most paths were being inherited from the node manager (mini-YARN cluster). > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15070370#comment-15070370 ] Sangjin Lee commented on MAPREDUCE-6577: After looking into the Windows side of things for a while, I realize that there are bigger issues to address on that side that go beyond the scope of this JIRA. Also, Windows isn't exactly broken in the same manner as Linux is by MAPREDUCE-6021 as it didn't really touch Windows. I filed a separate JIRA to address the Windows parity and less major issues (MAPREDUCE-6588). I propose we tackle the Windows aspect in that JIRA. I'll update the patch shortly to preserve the Windows behavior as it is today. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6588) default values for PATH for tasks are not quite correct
[ https://issues.apache.org/jira/browse/MAPREDUCE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15070365#comment-15070365 ] Sangjin Lee commented on MAPREDUCE-6588: One interesting observation I made while trying to use the cross-platform macros for the MR AM as a test: {noformat} PATH={{PATH}};{{HADOOP_COMMON_HOME}}\bin {noformat} The node manager sets the following line in the batch script that launches the AM: {noformat} @set PATH=%PATH%;%HADOOP_COMMON_HOME%\bin {noformat} However, on the AM process, the PATH environment variable is as follows: {noformat} PATH=%PATH%;%HADOOP_COMMON_HOME%\bin;C:\hadoop\hadoop-mapreduce-project\hadoop-mapreduce-client\hadoop-mapreduce-client-jobclient/../../../hadoop-common-project/hadoop-common/target\bin {noformat} While the substitution for HADOOP_COMMON_HOME is made, PATH is not. And also the original macro is preserved instead of being replaced. Maybe this is because I'm not familiar with Windows batch scripting, but this causes the existing path value to be dropped. > default values for PATH for tasks are not quite correct > --- > > Key: MAPREDUCE-6588 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6588 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee > > There are several issues with the way PATH are handled for MR tasks and AMs > on Windows. > - it does not get the PWD as part of the path, unlike the non-Windows case > (see MAPREDUCE-6021) > - the MR AM doesn't get the Hadoop's bin directory in the path explicitly > through mapred configuration; rather, it implicitly inherits it from the node > manager (see MAPREDUCE-6577) > - the default value for the MR task PATH uses "%...%" macros which may be > expanded on the client side prematurely; this is probably OK for tasks (as > it's done by the AM), but not OK for AMs; the late-binding cross-platform > macros should be used > I took a stab at addressing these as part of MAPREDUCE-6577, but I realized > that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus, > I'm filing a separate JIRA to address the Windows side of things. > On a related note, there are quite a few brittle tests on testing these > (TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such > a way that the tests pass against the current behavior but don't provide a > lot of robust test value. We should also update those tests to make them more > robust. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6588) default values for PATH for tasks are not quite correct
Sangjin Lee created MAPREDUCE-6588: -- Summary: default values for PATH for tasks are not quite correct Key: MAPREDUCE-6588 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6588 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.6.0 Reporter: Sangjin Lee There are several issues with the way PATH are handled for MR tasks and AMs on Windows. - it does not get the PWD as part of the path, unlike the non-Windows case (see MAPREDUCE-6021) - the MR AM doesn't get the Hadoop's bin directory in the path explicitly through mapred configuration; rather, it implicitly inherits it from the node manager (see MAPREDUCE-6577) - the default value for the MR task PATH uses "%...%" macros which may be expanded on the client side prematurely; this is probably OK for tasks (as it's done by the AM), but not OK for AMs; the late-binding cross-platform macros should be used I took a stab at addressing these as part of MAPREDUCE-6577, but I realized that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus, I'm filing a separate JIRA to address the Windows side of things. On a related note, there are quite a few brittle tests on testing these (TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such a way that the tests pass against the current behavior but don't provide a lot of robust test value. We should also update those tests to make them more robust. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068539#comment-15068539 ] Sangjin Lee commented on MAPREDUCE-6577: Thanks Chris. I actually stated down my path on setting that up, but I was having tons of trouble setting up the Visual Studio (or Windows SDK?). Even with the native build turned off, I kept getting errors around msbuild and Visual Studio. I tried all kinds of different versions (10, 2013, 2015) with no success. I'll give it one more shot. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067641#comment-15067641 ] Sangjin Lee commented on MAPREDUCE-6577: No worries. Thanks for checking. It's rather difficult to acquire a Windows machine that's ready to build hadoop. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066942#comment-15066942 ] Sangjin Lee commented on MAPREDUCE-6577: Thanks [~laurentgo] and [~cnauroth] for the pointers! Yes, I am aware that Mac uses DYLD_LIBRARY_PATH, but there seem to be more quirks (perhaps version-dependent?) that mess with that. Since it's an existing larger issue, I think it's safe to say it's outside the scope of this JIRA. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066815#comment-15066815 ] Sangjin Lee commented on MAPREDUCE-6577: The latest patch (v.3) should now be good to review. I was thrown off for a bit because Mac OS (on which I was testing some of these unit tests) doesn't seem to handle LD_LIBRARY_PATH in the same way as Linux. In fact, the trunk version of TestMiniMRChildTask does not pass on Mac. But on Linux, the test is fine before and after the patch. I would appreciate your review. Thanks! > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Attachment: MAPREDUCE-6577.03.patch Posted patch v.3. Fixed TestMiniMRChildTask#testMapRedExecutionEnv(). Removed the first part of the test as it is not meaningful to test the HADOOP_COMMON_HOME substitution as it is no longer substituted with the late binding. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064950#comment-15064950 ] Sangjin Lee commented on MAPREDUCE-6577: It looks like TestMiniMRChildTask#testMapRedExecutionEnv() needs updating. Please hold off on committing it until I update the patch. Thanks. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Attachment: MAPREDUCE-6577.02.patch Posted patch v.2 that addresses Jason's review comments. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064196#comment-15064196 ] Sangjin Lee commented on MAPREDUCE-6577: Thanks for the clarification. Yes, that's what I'm going to do now that the variables are not going to be expanded. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064181#comment-15064181 ] Sangjin Lee commented on MAPREDUCE-6577: Thanks for the pointer [~jlowe]! MRApps.crossPlatformifyMREnv() does that only if mapreduce.app-submission.cross-platform is true (which is false by default)? So it looks like it'd be safer if I used \{\{ and \}\} directly, no? Also, the method requires a Configuration object and an Environment object, so I'm not sure if that works in MRJobConfig. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063442#comment-15063442 ] Sangjin Lee commented on MAPREDUCE-6577: All the issues that are flagged by jenkins (findbugs, javac, checkstyle, and unit tests) appear to be existing issues that are unrelated to the patch. I'd appreciate your review on the patch. Thanks! > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Status: Patch Available (was: Open) Posted patch v.1. Added the default for the AM admin user env (which has the same value as the non-AM admin user env). Also did a small amount of yak-shaving. I suspect that the lib path handling on Windows is not quite right (PWD was not being added for Windows). I tried to make it a little more cross-platform there. Folks who use Windows ([~cnauroth]?), I'd greatly appreciate it if you could review the patch from that perspective. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Attachment: MAPREDUCE-6577.01.patch > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061631#comment-15061631 ] Sangjin Lee commented on MAPREDUCE-6577: My proposal would be to handle this in the same way as the non-AM tasks; i.e. provide the same default value for yarn.app.mapreduce.am.admin.user.env. That would also complete the intent of MAPREDUCE-6021. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)