[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199540#comment-16199540
 ] 

Sangjin Lee commented on MAPREDUCE-5951:


+1. Thanks for the great work and taking it to completion [~ctrezzo]!

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, 
> MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch, 
> MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, 
> MAPREDUCE-5951-trunk-v15.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch, MAPREDUCE-5951-trunk.016.patch, 
> MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch, 
> MAPREDUCE-5951-trunk.019.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957590#comment-15957590
 ] 

Sangjin Lee commented on MAPREDUCE-6846:


Thanks [~dan...@cloudera.com]! Would you do the honors of committing this?

> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch, 
> MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch, 
> MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch, 
> MAPREDUCE-6846-trunk.006.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser, 
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt 
> does not exist
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
>   at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
>   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here 
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute 
> path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e. 
> test.txt#frag.txt) ||
> || -libjars | FileNotFound | FileNotFound|FileNotFound|
> || -files | (/) | (/) | (/) |
> || -archives | (/) | (/) | (/) |



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954409#comment-15954409
 ] 

Sangjin Lee commented on MAPREDUCE-6846:


The latest patch LGTM. [~dan...@cloudera.com] what do you think?

> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch, 
> MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch, 
> MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch, 
> MAPREDUCE-6846-trunk.006.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser, 
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt 
> does not exist
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
>   at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
>   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here 
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute 
> path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e. 
> test.txt#frag.txt) ||
> || -libjars | FileNotFound | FileNotFound|FileNotFound|
> || -files | (/) | (/) | (/) |
> || -archives | (/) | (/) | (/) |



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Assigned] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned MAPREDUCE-6732:
--

Assignee: Vrushali C  (was: Sangjin Lee)

> mapreduce tasks for YARN Timeline Service v.2: alpha 2
> --
>
> Key: MAPREDUCE-6732
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>
> This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline 
> Service v.2 alpha 2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based 
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any 
> subtask work on this JIRA will be committed to those 2 branches.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922870#comment-15922870
 ] 

Sangjin Lee commented on MAPREDUCE-6846:


bq. I took a look through various branches and see that it has been broken at 
least back to branch-2.0.0-alpha. It seems that this has been broken for a very 
long time.

That's interesting. We saw that terasort uses fragments. But I don't think it 
has been broken that long?

> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser, 
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt 
> does not exist
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
>   at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
>   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here 
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute 
> path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e. 
> test.txt#frag.txt) ||
> || -libjars | FileNotFound | FileNotFound|FileNotFound|
> || -files | (/) | (/) | (/) |
> || -archives | (/) | (/) | (/) |



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922622#comment-15922622
 ] 

Sangjin Lee commented on MAPREDUCE-6846:


Thanks [~ctrezzo] for your patch! I'm generally fine with the approach taken in 
this patch. Just curious, was this always broken, or is it a regression? If the 
latter, could you add the JIRA that introduced the regression?

(JobResourceUploader.java)
- l.169: {{getPathURI()}} will always result in a URL with a fragment even if 
the original path did not have the fragment. This is not incorrect, but this 
would have an effect of bloating the distributed cache entries in the 
configuration which can be sizable in cases. Can we not add the fragment if the 
fragment is null and the original URL does not have the fragment either?
- l.171: nit: it can simply be {{foundFragment = tmpURI.getFragment() != null}}
- I'm wondering, would it lead to simpler code if we iterate over the list once 
to determine whether there is a fragment and iterate again to do the population 
of the distributed cache? It might be slightly more expensive, but may lead to 
code that's easier to understand/maintain. Thoughts?


> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser, 
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt 
> does not exist
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
>   at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
>   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior 

[jira] [Commented] (MAPREDUCE-6855) Specify charset when create String in CredentialsTestJob

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894642#comment-15894642
 ] 

Sangjin Lee commented on MAPREDUCE-6855:


Thanks.

> Specify charset when create String in CredentialsTestJob
> 
>
> Key: MAPREDUCE-6855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6855
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Minor
>  Labels: newbie
>
> {code}
>   String secretValueStr = new String (secretValue);
> {code}
> should be
> {code}
>   String secretValueStr = new String(secretValue, StandardCharsets.UTF_8);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6855) Specify charset when create String in CredentialsTestJob

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894615#comment-15894615
 ] 

Sangjin Lee commented on MAPREDUCE-6855:


It would be great if we can catch things like this with a findbugs rule. Do you 
know if that is straightforward?

> Specify charset when create String in CredentialsTestJob
> 
>
> Key: MAPREDUCE-6855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6855
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Minor
>  Labels: newbie
>
> {code}
>   String secretValueStr = new String (secretValue);
> {code}
> should be
> {code}
>   String secretValueStr = new String(secretValue, StandardCharsets.UTF_8);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-5951) Add support for the YARN Shared Cache

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-5951:
---
Status: Open  (was: Patch Available)

Just to reflect the current status (waiting for an updated patch). Let me know 
if that is not accurate. Thanks!

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860182#comment-15860182
 ] 

Sangjin Lee commented on MAPREDUCE-6846:


Thanks for finding this issue [~ctrezzo]. Personally I would prefer option #1...

> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
>
> If a user specifies a fragment for a libjars path via generic options parser, 
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt 
> does not exist
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
>   at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
>   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at 
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here 
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute 
> path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e. 
> test.txt#frag.txt) ||
> || -libjars | FileNotFound | FileNotFound|FileNotFound|
> || -files | (/) | (/) | (/) |
> || -archives | (/) | (/) | (/) |



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6836) exception thrown when accessing the job configuration web UI

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6836:
--

 Summary: exception thrown when accessing the job configuration web 
UI
 Key: MAPREDUCE-6836
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6836
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: webapps
Affects Versions: 3.0.0-alpha2
Reporter: Sangjin Lee
Priority: Minor


When I navigate the MR job web UI and click the configuration link, the AM 
shows an exception:
{noformat}
2017-01-25 11:40:55,521 ERROR [qtp2126664214-26] 
org.apache.hadoop.yarn.webapp.Dispatcher: error handling URI: /mapreduc
e/conf/job_1485372765455_0002
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at 
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at 
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at 
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at 
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at 
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at 
com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:179)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1458)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:524)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at 

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817206#comment-15817206
 ] 

Sangjin Lee commented on MAPREDUCE-5951:


I went over the patch with Chris in some detail today, and am posting the 
review comments here for the record.

(MRJobConfig.java)
- {{mapreduce.job.jobjar.visibility}} and 
{{mapreduce.job.jobjar.sharedcache.uploadpolicy}} are computed values that are 
not user-facing; in that case we should not even define the defaults so that 
there is no confusion that these values are computed

(JobResourceUploader.java)
- l.159-160: I understand lines 161-165 are there to support programmatic use 
cases of the distributed cache that come in outside of the job submitter code 
path. Can we make the comments clearer so that the intent of this comes 
through? We could also annotate them in {{MRJobConfig}}.
- l.171-172: it might be slightly better to use {{LinkedHashMap}}. That way, 
we'd have a predictable iteration order (the order in which they are specified 
in the user values).
- l.219: To be fair, this is a bug we need to fix, right? Then can we file a 
JIRA and add the JIRA id here?
- l.237-240: I would not worry about handling previous values here. Having 
duplicate paths is not really supported and the worst case scenario here is to 
reset this upload policy with the same value.
- l.260-263: I think we can improve on this, and reconcile the shared cache 
with the wildcard feature. We could see if any resource is uploaded to the 
staging directory, and if so, still preserve the wildcard entry. We also need 
to consider the case where the shared cache is disabled but the wildcard is 
enabled.
- l.291-294: same comment as above
- l.348-351: same comment as above
- l.388: Since we're dealing with a local filesystem URI based on l.381-382, 
the authority check is not meaningful. We should remove this check.
- On a larger note, the path/URI handling between the job jar, libjars, files, 
and archives is not very consistent, which is an existing behavior. We need to 
see if they need to get the same consistent treatment for this to work.

(Job.java)
- l.1446: I realized later that passing an empty map has an effect of nulling 
out the config value; perhaps we could make that more explicit in the javadoc 
and/or comments/code?
- l.1449-1463: nit: it might be slightly easier to read by using a simple 
string concatenation with "+" (JVM internally uses the {{StringBuilder}})
- l.1490: here also it might be better to use {{LinkedHashMap}}


> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-trunk-v1.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
> MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
> MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
> MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6818:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: YARN-5355
   Status: Resolved  (was: Patch Available)

Committed the patch to YARN-5355 and YARN-5355-branch-2. Thanks [~gtCarrera9] 
for your contribution!

> Remove direct reference to TimelineClientImpl
> -
>
> Key: MAPREDUCE-6818
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Li Lu
>Assignee: Li Lu
>  Labels: newbie++
> Fix For: YARN-5355
>
> Attachments: YARN-5974-YARN-5355.001.patch
>
>
> [~sjlee0]'s quick audit shows that things that are referencing 
> TimelineClientImpl directly today:
> JobHistoryFileReplayMapperV1 (MR)
> SimpleEntityWriterV1 (MR)
> TestDistributedShell (DS)
> TestDSAppMaster (DS)
> TestNMTimelinePublisher (node manager)
> TestTimelineWebServicesWithSSL (AHS)
> This is not the right way to use TimelineClient and we should avoid direct 
> reference to TimelineClientImpl as much as possible. 
> Any newcomers to the community are more than welcome to take this. If this 
> remains unassigned for ~24hrs I'll jump in and do a quick fix. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6818:
---
Target Version/s: YARN-5355

> Remove direct reference to TimelineClientImpl
> -
>
> Key: MAPREDUCE-6818
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Li Lu
>Assignee: Li Lu
>  Labels: newbie++
> Attachments: YARN-5974-YARN-5355.001.patch
>
>
> [~sjlee0]'s quick audit shows that things that are referencing 
> TimelineClientImpl directly today:
> JobHistoryFileReplayMapperV1 (MR)
> SimpleEntityWriterV1 (MR)
> TestDistributedShell (DS)
> TestDSAppMaster (DS)
> TestNMTimelinePublisher (node manager)
> TestTimelineWebServicesWithSSL (AHS)
> This is not the right way to use TimelineClient and we should avoid direct 
> reference to TimelineClientImpl as much as possible. 
> Any newcomers to the community are more than welcome to take this. If this 
> remains unassigned for ~24hrs I'll jump in and do a quick fix. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6818:
---
Issue Type: Sub-task  (was: Bug)
Parent: MAPREDUCE-6732

> Remove direct reference to TimelineClientImpl
> -
>
> Key: MAPREDUCE-6818
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Li Lu
>Assignee: Li Lu
>  Labels: newbie++
> Attachments: YARN-5974-YARN-5355.001.patch
>
>
> [~sjlee0]'s quick audit shows that things that are referencing 
> TimelineClientImpl directly today:
> JobHistoryFileReplayMapperV1 (MR)
> SimpleEntityWriterV1 (MR)
> TestDistributedShell (DS)
> TestDSAppMaster (DS)
> TestNMTimelinePublisher (node manager)
> TestTimelineWebServicesWithSSL (AHS)
> This is not the right way to use TimelineClient and we should avoid direct 
> reference to TimelineClientImpl as much as possible. 
> Any newcomers to the community are more than welcome to take this. If this 
> remains unassigned for ~24hrs I'll jump in and do a quick fix. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Moved] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee moved YARN-5974 to MAPREDUCE-6818:
--

Affects Version/s: (was: YARN-5355)
  Component/s: (was: timelineserver)
  Key: MAPREDUCE-6818  (was: YARN-5974)
  Project: Hadoop Map/Reduce  (was: Hadoop YARN)

> Remove direct reference to TimelineClientImpl
> -
>
> Key: MAPREDUCE-6818
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Li Lu
>Assignee: Li Lu
>  Labels: newbie++
> Attachments: YARN-5974-YARN-5355.001.patch
>
>
> [~sjlee0]'s quick audit shows that things that are referencing 
> TimelineClientImpl directly today:
> JobHistoryFileReplayMapperV1 (MR)
> SimpleEntityWriterV1 (MR)
> TestDistributedShell (DS)
> TestDSAppMaster (DS)
> TestNMTimelinePublisher (node manager)
> TestTimelineWebServicesWithSSL (AHS)
> This is not the right way to use TimelineClient and we should avoid direct 
> reference to TimelineClientImpl as much as possible. 
> Any newcomers to the community are more than welcome to take this. If this 
> remains unassigned for ~24hrs I'll jump in and do a quick fix. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626604#comment-15626604
 ] 

Sangjin Lee commented on MAPREDUCE-6704:


I am +1 for #3. There is the yuck factor of mixing up MR into YARN, but the 
issue is significant enough and the change small enough that I think we can 
make an exception here.

We can move the discussion to the mailing list to expand the audience.

> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch, 
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set 
> .After 
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}} 
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with 
> state FAILED due to: Application application_1462155939310_0004 failed 2 
> times due to AM Container for appattempt_1462155939310_0004_02 exited 
> with  exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; 
> support was removed in 8.0
> Error: Could not find or load main class 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; 
> support was removed in 8.0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6741) add MR support to redact job conf properties

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570269#comment-15570269
 ] 

Sangjin Lee commented on MAPREDUCE-6741:


According to [the discussion|https://s.apache.org/Qu1g] we were having on the 
release cadence and EOL, the 2.6.x line could be EOLed some time in Nov. 2016. 
Although it is far from being concluded, the fate of another 2.6 release is at 
least uncertain. Why don't we hold off on backporting it to 2.6?

We should probably raise the release cadence and EOL topic again and conclude 
it at some point...

> add MR support to redact job conf properties
> 
>
> Key: MAPREDUCE-6741
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
> Attachments: mapreduce6741.001.patch, mapreduce6741.002.patch, 
> mapreduce6741.003.patch, mapreduce6741.004.patch, mapreduce6741.005.patch, 
> mapreduce6741.006.patch, mapreduce6741.branch-2.8.patch
>
>
> JHS today displays all Job conf properties in Web UI directly. Users may have 
> some credentials or any sensitive information they added to the job conf but 
> do not want to be shown in Web UI. It'd be nice if we can allow users to 
> specify a set of properties which JHS will filter out when Job conf is 
> displayed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6779) Mapreduce job failure on submission

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee resolved MAPREDUCE-6779.

Resolution: Resolved
  Assignee: Sangjin Lee

> Mapreduce job failure on submission
> ---
>
> Key: MAPREDUCE-6779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6779
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Sangjin Lee
>Priority: Blocker
>
> Configure Hibench
> Try running Enhanced TestDFSIO
> {noformat}
> 2016-09-15 18:20:24,849 INFO mapreduce.Job: Task Id : 
> attempt_1473943118844_0001_m_01_0, Status : FAILED
> Error: java.lang.RuntimeException: Error in configuring object
> at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
> at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
> at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1806)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> ... 9 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2330)
> at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1108)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
> ... 14 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> Class org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2298)
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2322)
> ... 16 more
> Caused by: java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
> at 
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2202)
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2296)
> ... 17 more
> {noformat}
> *mapreduce.JobResourceUploader: No job jar file set.  User classes may not be 
> found. See Job or Job#setJar(String).*
> Job jar is not getting set and not uploaded to staging dir



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6779) Mapreduce job failure on submission

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15493447#comment-15493447
 ] 

Sangjin Lee commented on MAPREDUCE-6779:


Can you provide the hadoop jar command line with which you see this issue? Does 
it happen with any job submission or only with this job?

> Mapreduce job failure on submission
> ---
>
> Key: MAPREDUCE-6779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6779
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
>
> Configure Hibench
> Try running Enhanced TestDFSIO
> {noformat}
> 2016-09-15 18:20:24,849 INFO mapreduce.Job: Task Id : 
> attempt_1473943118844_0001_m_01_0, Status : FAILED
> Error: java.lang.RuntimeException: Error in configuring object
> at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
> at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
> at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1806)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> ... 9 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2330)
> at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1108)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
> ... 14 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> Class org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2298)
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2322)
> ... 16 more
> Caused by: java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
> at 
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2202)
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2296)
> ... 17 more
> {noformat}
> *mapreduce.JobResourceUploader: No job jar file set.  User classes may not be 
> found. See Job or Job#setJar(String).*
> Job jar is not getting set and not uploaded to staging dir



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6637) Testcase Failure : TestFileInputFormat.testSplitLocationInfo

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6637:
---
Fix Version/s: 2.6.5

Cherry-picked it to 2.6.5 (trivial).

> Testcase Failure : TestFileInputFormat.testSplitLocationInfo
> 
>
> Key: MAPREDUCE-6637
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6637
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.7.3, 2.6.5, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6637.patch
>
>
> Following testcase is failing after HADOOP-12810
> {noformat}
> FAILED:  org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo[0]
> Error Message:
> expected:<2> but was:<1>
> Stack Trace:
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo(TestFileInputFormat.java:115)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-5817) Mappers get rescheduled on node transition even after all reducers are completed

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-5817:
---
Fix Version/s: 2.6.5

Cherry-picked it into 2.6.5.

> Mappers get rescheduled on node transition even after all reducers are 
> completed
> 
>
> Key: MAPREDUCE-5817
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5817
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.3.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.7.3, 2.6.5, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5817.001.patch, MAPREDUCE-5817.002.patch, 
> mapreduce-5817.patch
>
>
> We're seeing a behavior where a job runs long after all reducers were already 
> finished. We found that the job was rescheduling and running a number of 
> mappers beyond the point of reducer completion. In one situation, the job ran 
> for some 9 more hours after all reducers completed!
> This happens because whenever a node transition (to an unusable state) comes 
> into the app master, it just reschedules all mappers that already ran on the 
> node in all cases.
> Therefore, if any node transition has a potential to extend the job period. 
> Once this window opens, another node transition can prolong it, and this can 
> happen indefinitely in theory.
> If there is some instability in the pool (unhealthy, etc.) for a duration, 
> then any big job is severely vulnerable to this problem.
> If all reducers have been completed, JobImpl.actOnUnusableNode() should not 
> reschedule mapper tasks. If all reducers are completed, the mapper outputs 
> are no longer needed, and there is no need to reschedule mapper tasks as they 
> would not be consumed anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6365) Refactor JobResourceUploader#uploadFilesInternal

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6365:
---
Fix Version/s: (was: 3.0.0-alpha2)
   2.9.0

> Refactor JobResourceUploader#uploadFilesInternal
> 
>
> Key: MAPREDUCE-6365
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6365
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: MAPREDUCE-6365-trunk-v1.patch
>
>
> JobResourceUploader#uploadFilesInternal is a large method and there are 
> similar pieces of code that could probably be pulled out into separate 
> methods.  This refactor would improve readability of the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6365) Refactor JobResourceUploader#uploadFilesInternal

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6365:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   Status: Resolved  (was: Patch Available)

Committed the patch to trunk. Thanks [~ctrezzo] for your contribution! Should 
this be backported to an earlier version? Branch-2 (2.9.0)?

> Refactor JobResourceUploader#uploadFilesInternal
> 
>
> Key: MAPREDUCE-6365
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6365
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6365-trunk-v1.patch
>
>
> JobResourceUploader#uploadFilesInternal is a large method and there are 
> similar pieces of code that could probably be pulled out into separate 
> methods.  This refactor would improve readability of the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6365) Refactor JobResourceUploader#uploadFilesInternal

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385168#comment-15385168
 ] 

Sangjin Lee commented on MAPREDUCE-6365:


+1. Will commit soon.

> Refactor JobResourceUploader#uploadFilesInternal
> 
>
> Key: MAPREDUCE-6365
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6365
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6365-trunk-v1.patch
>
>
> JobResourceUploader#uploadFilesInternal is a large method and there are 
> similar pieces of code that could probably be pulled out into separate 
> methods.  This refactor would improve readability of the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6735) Performance degradation caused by MAPREDUCE-5465 and HADOOP-12107

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384866#comment-15384866
 ] 

Sangjin Lee commented on MAPREDUCE-6735:


It is certainly surprising that HADOOP-12107 is making a difference on 
terasort. What version of hadoop are you using for your test? Java version? Is 
it repeatable (i.e. the gap shows up consistently)?

FYI, the nature of HADOOP-12107 has to do with *when* to clean up a certain 
data ( {{allData}} ) inside the {{FileSystem.Statistics}} objects. Before this 
change, it would get cleaned up when the owner thread gets garbage collected 
*and* a read operation is done on the {{Statistics}} object. By read operations 
I mean methods such as {{getBytesRead()}} and so on.

After this change, the timing of this clean-up no longer depends on the read 
operations, and it will be done promptly when the thread is garbage collected. 
So in a sense, the change first ensures there is clean-up no matter what, and 
also moves up the timing of the clean-up.

The worst-case scenario in which this can have a negative impact on performance 
is if the use case *never* reads the statistics. Prior to the change, as long 
as the heap can contain these objects, no clean-up will be done. With the 
change now we do perform additional clean-up on threads garbage collection.

A subsequent observation is that the impact of the clean-up is greater if there 
is a *high degree of thread churn* within the JVM. If we're talking about only 
a handful of threads or long-lived threads, there should really be no 
difference.

I would greatly appreciate it if you could dig a little deeper via logging or 
low overhead profiling to pinpoint the correlation. Thanks.

> Performance degradation caused by MAPREDUCE-5465 and HADOOP-12107
> -
>
> Key: MAPREDUCE-6735
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6735
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Alexandr Balitsky
>
> Two commits, MAPREDUCE-5465 and HADOOP-12107 are making Terasort on YARN 10% 
> slower.
> Reduce phase with those commits ~5 mins
> Reduce phase without ~3.5 mins
> Average Reduce is taking 4mins, 16sec with those commits compared to 3mins, 
> 48sec without.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6731:
---
Attachment: MAPREDUCE-6731.02.patch

Posted patch v.2. A small change in the directory name (to make it shorter).

> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail 
> for concurrent tests
> -
>
> Key: MAPREDUCE-6731
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: MAPREDUCE-6731.01.patch, MAPREDUCE-6731.02.patch
>
>
> {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses 
> the default file-system storage directory, and is brittle against concurrent 
> tests.
> We should use a unique storage directory for the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371681#comment-15371681
 ] 

Sangjin Lee commented on MAPREDUCE-6731:


The test failure is unrelated. I would greatly appreciate your review. Thanks!

> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail 
> for concurrent tests
> -
>
> Key: MAPREDUCE-6731
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: MAPREDUCE-6731.01.patch
>
>
> {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses 
> the default file-system storage directory, and is brittle against concurrent 
> tests.
> We should use a unique storage directory for the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371426#comment-15371426
 ] 

Sangjin Lee commented on MAPREDUCE-6732:


I have just created and pushed branches {{YARN-5355}} and 
{{YARN-5355-branch-2}}. They are now open for this feature development.

> mapreduce tasks for YARN Timeline Service v.2: alpha 2
> --
>
> Key: MAPREDUCE-6732
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>
> This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline 
> Service v.2 alpha 2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based 
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any 
> subtask work on this JIRA will be committed to those 2 branches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6732:
---
Description: 
This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline 
Service v.2 alpha 2.

This is developed on feature branches: {{YARN-5355}} for the trunk-based 
development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any 
subtask work on this JIRA will be committed to those 2 branches.

  was:This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline 
Service v.2.


> mapreduce tasks for YARN Timeline Service v.2: alpha 2
> --
>
> Key: MAPREDUCE-6732
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>
> This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline 
> Service v.2 alpha 2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based 
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any 
> subtask work on this JIRA will be committed to those 2 branches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6731:
---
Status: Patch Available  (was: In Progress)

> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail 
> for concurrent tests
> -
>
> Key: MAPREDUCE-6731
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: MAPREDUCE-6731.01.patch
>
>
> {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses 
> the default file-system storage directory, and is brittle against concurrent 
> tests.
> We should use a unique storage directory for the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6731:
---
Attachment: MAPREDUCE-6731.01.patch

Posted patch v.1.

This uses a test directory local to the build, instead of the global shared 
storage directory. I also made the path separator cross-platform.

> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail 
> for concurrent tests
> -
>
> Key: MAPREDUCE-6731
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: MAPREDUCE-6731.01.patch
>
>
> {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses 
> the default file-system storage directory, and is brittle against concurrent 
> tests.
> We should use a unique storage directory for the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Work started] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on MAPREDUCE-6731 started by Sangjin Lee.
--
> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail 
> for concurrent tests
> -
>
> Key: MAPREDUCE-6731
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>
> {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses 
> the default file-system storage directory, and is brittle against concurrent 
> tests.
> We should use a unique storage directory for the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6318) Refactor JobHistoryEventHandler for handling events in ATS v1 after YARN-2928 merge back to trunk and branch-2

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6318:
---
Parent Issue: MAPREDUCE-6732  (was: MAPREDUCE-6331)

> Refactor JobHistoryEventHandler for handling events in ATS v1 after YARN-2928 
> merge back to trunk and branch-2 
> ---
>
> Key: MAPREDUCE-6318
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6318
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
>
> Per discussion in YARN-3046, we need to refactor the process flow for 
> timeline events (ATS v1) in JobHistoryEventHandler. The refactor work should 
> get happen after YARN-2928 merged into trunk and branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6732:
--

 Summary: mapreduce tasks for YARN Timeline Service v.2: alpha 2
 Key: MAPREDUCE-6732
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Sangjin Lee
Assignee: Sangjin Lee


This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline 
Service v.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6731:
--

 Summary: 
TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail 
for concurrent tests
 Key: MAPREDUCE-6731
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0-alpha1
Reporter: Sangjin Lee
Assignee: Sangjin Lee


{{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses 
the default file-system storage directory, and is brittle against concurrent 
tests.

We should use a unique storage directory for the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6331) [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee resolved MAPREDUCE-6331.

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha1

It's been merged to trunk. Huge thanks to the contributors who worked on this 
feature ([~jrottinghuis], [~djp], [~gtCarrera9], [~Naganarasimha], 
[~varun_saxena], [~vinodkv], [~vrushalic], and [~zjshen]) and everyone who 
participated in reviews and feedback!

> [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)
> 
>
> Key: MAPREDUCE-6331
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6331
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Sangjin Lee
> Fix For: 3.0.0-alpha1
>
>
> Tracking umbrella for all MR changes to make it work with Timeline Service 
> Nextgen - YARN-2928.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343119#comment-15343119
 ] 

Sangjin Lee commented on MAPREDUCE-6720:


Please feel free to go ahead and commit it. Thanks!

> Inconsistent values of counters across tasks and job reported to timeline 
> service.
> --
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: Counters For Job.png, Counters for Tasks.png, 
> MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch, 
> MAPREDUCE-6720-YARN-2928.03.patch, MAPREDUCE-6720-YARN-2928.04.patch
>
>
> While testing found below issue. For some of the task counters, we do not 
> have consistent values. This is not the case with every counter though.
> Consider the case of counter 
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps 
> associated with this flow run, the values were 312006 bytes each (which 
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further 
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2 
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes 
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) 
> or 623842 bytes but it is only 312006 bytes which indicates that only counter 
> value of mappers is being picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342657#comment-15342657
 ] 

Sangjin Lee commented on MAPREDUCE-6720:


The latest patch looks good to me. Please let us know if you're OK with the 
approach and the patch. Thanks!

> Inconsistent values of counters across tasks and job reported to timeline 
> service.
> --
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: Counters For Job.png, Counters for Tasks.png, 
> MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch, 
> MAPREDUCE-6720-YARN-2928.03.patch, MAPREDUCE-6720-YARN-2928.04.patch
>
>
> While testing found below issue. For some of the task counters, we do not 
> have consistent values. This is not the case with every counter though.
> Consider the case of counter 
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps 
> associated with this flow run, the values were 312006 bytes each (which 
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further 
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2 
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes 
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) 
> or 623842 bytes but it is only 312006 bytes which indicates that only counter 
> value of mappers is being picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342402#comment-15342402
 ] 

Sangjin Lee commented on MAPREDUCE-6720:


The latest patch looks good to me, pending the checkstyle and whitespace fixes 
and the jenkins run. Again, let's hold off until the rebase is complete before 
committing this.

> Inconsistent values of counters across tasks and job reported to timeline 
> service.
> --
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: Counters For Job.png, Counters for Tasks.png, 
> MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch, 
> MAPREDUCE-6720-YARN-2928.03.patch
>
>
> While testing found below issue. For some of the task counters, we do not 
> have consistent values. This is not the case with every counter though.
> Consider the case of counter 
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps 
> associated with this flow run, the values were 312006 bytes each (which 
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further 
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2 
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes 
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) 
> or 623842 bytes but it is only 312006 bytes which indicates that only counter 
> value of mappers is being picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6719) The list of -libjars archives should be replaced with a wildcard in the distributed cache to reduce the application footprint in the state store

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6719:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
   Status: Resolved  (was: Patch Available)

Committed patch v.2 to trunk and branch-2 (2.9.0). Thanks [~templedf] for your 
contribution!

> The list of -libjars archives should be replaced with a wildcard in the 
> distributed cache to reduce the application footprint in the state store
> 
>
> Key: MAPREDUCE-6719
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: MAPREDUCE-6719.001.patch, MAPREDUCE-6719.002.patch
>
>
> When using the -libjars option to add classes to the classpath, every library 
> so added is explicitly listed in the ContainerLaunchContext's local resources 
> even though they're all uploaded to the same directory in HDFS. When using 
> tools like Crunch without an uber JAR or when trying to take advantage of the 
> shared cache, the number of libraries can be quite large. We've seen many 
> cases where we had to turn down the max number of applications to prevent ZK 
> from running out of heap because of the size of the state store entries.
> This JIRA proposes to allow for wildcards both in the internal processing of 
> the -libjars switch and in paths added through the Job and DistributedCache 
> classes. Rather than listing all files independently, this JIRA proposes to 
> replace the complete list of libdir files with the wildcarded libdir 
> directory, e.g. "libdir/*". This behavior is the same as the current behavior 
> when using -libjars, but avoids explicitly listing every file.
> This capability will also be exposed by the 
> {{DistributedCache.addCacheFile()}} method.
> See YARN-4958 for the NM side of the implementation and additional discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6719) The list of -libjars archives should be replaced with a wildcard in the distributed cache to reduce the application footprint in the state store

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342378#comment-15342378
 ] 

Sangjin Lee commented on MAPREDUCE-6719:


Thanks Daniel! Will commit it shortly.

> The list of -libjars archives should be replaced with a wildcard in the 
> distributed cache to reduce the application footprint in the state store
> 
>
> Key: MAPREDUCE-6719
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: MAPREDUCE-6719.001.patch, MAPREDUCE-6719.002.patch
>
>
> When using the -libjars option to add classes to the classpath, every library 
> so added is explicitly listed in the ContainerLaunchContext's local resources 
> even though they're all uploaded to the same directory in HDFS. When using 
> tools like Crunch without an uber JAR or when trying to take advantage of the 
> shared cache, the number of libraries can be quite large. We've seen many 
> cases where we had to turn down the max number of applications to prevent ZK 
> from running out of heap because of the size of the state store entries.
> This JIRA proposes to allow for wildcards both in the internal processing of 
> the -libjars switch and in paths added through the Job and DistributedCache 
> classes. Rather than listing all files independently, this JIRA proposes to 
> replace the complete list of libdir files with the wildcarded libdir 
> directory, e.g. "libdir/*". This behavior is the same as the current behavior 
> when using -libjars, but avoids explicitly listing every file.
> This capability will also be exposed by the 
> {{DistributedCache.addCacheFile()}} method.
> See YARN-4958 for the NM side of the implementation and additional discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342368#comment-15342368
 ] 

Sangjin Lee commented on MAPREDUCE-6720:


[~varun_saxena], could you look at the checkstyle and whitespace issues? Thanks!

> Inconsistent values of counters across tasks and job reported to timeline 
> service.
> --
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: Counters For Job.png, Counters for Tasks.png, 
> MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch, 
> MAPREDUCE-6720-YARN-2928.03.patch
>
>
> While testing found below issue. For some of the task counters, we do not 
> have consistent values. This is not the case with every counter though.
> Consider the case of counter 
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps 
> associated with this flow run, the values were 312006 bytes each (which 
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further 
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2 
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes 
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) 
> or 623842 bytes but it is only 312006 bytes which indicates that only counter 
> value of mappers is being picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342237#comment-15342237
 ] 

Sangjin Lee commented on MAPREDUCE-6720:


If we are going to store the map counters and reduce counters, we should make 
the metric names such that one can easily query for them only. So in that 
sense, making it a total prefix or suffix to the metric name might be a better 
idea? In other words, how about something like

{noformat}
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for total.
MAP:org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for MAP.
REDUCE:org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for REDUCE.
{noformat}

or 

{noformat}
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for total.
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS:MAP for MAP.
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS:REDUCE for REDUCE.
{noformat}
?

> Inconsistent values of counters across tasks and job reported to timeline 
> service.
> --
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: Counters For Job.png, Counters for Tasks.png, 
> MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch
>
>
> While testing found below issue. For some of the task counters, we do not 
> have consistent values. This is not the case with every counter though.
> Consider the case of counter 
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps 
> associated with this flow run, the values were 312006 bytes each (which 
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further 
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2 
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes 
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) 
> or 623842 bytes but it is only 312006 bytes which indicates that only counter 
> value of mappers is being picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342076#comment-15342076
 ] 

Sangjin Lee commented on MAPREDUCE-6720:


Thanks for reporting this issue [~varun_saxena]. It's a good catch. This 
reminded me of the same situation I encountered when I was looking at the job 
history file replaying in the performance test 
(https://github.com/apache/hadoop/blob/YARN-2928/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV2.java#L101).

I am OK with the recording only the total counters. If we do want the mapper 
counters and reducer counters too, we could consider prefixing or suffixing the 
metric names to disambiguate them.

> Inconsistent values of counters across tasks and job reported to timeline 
> service.
> --
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: Counters For Job.png, Counters for Tasks.png, 
> MAPREDUCE-6720-YARN-2928.01.patch
>
>
> While testing found below issue. For some of the task counters, we do not 
> have consistent values. This is not the case with every counter though.
> Consider the case of counter 
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps 
> associated with this flow run, the values were 312006 bytes each (which 
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further 
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2 
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes 
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) 
> or 623842 bytes but it is only 312006 bytes which indicates that only counter 
> value of mappers is being picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (MAPREDUCE-6719) -libjars should use wildcards to reduce the application footprint in the state store

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340813#comment-15340813
 ] 

Sangjin Lee edited comment on MAPREDUCE-6719 at 6/21/16 1:03 AM:
-

The latest patch looks good to me. The unit test that timed out is known to 
happen occasionally (can't locate an open JIRA for that one though). The 
deprecation warning is also an unfortunate result of {{DistributedCache}} being 
deprecated.

I do have a small quibble with the title of the JIRA. The title may give you an 
impression that with this JIRA one could use a wildcard in specifying the 
libjars *argument*. But that's not the case. What this JIRA does (in 
conjunction with YARN-4958) is to collapse the libjar directory in the 
configuration as well as the container launch context. But the current title 
can be confused with HADOOP-12747 ("support wildcard in libjars argument"). Can 
we modify the title to better describe this JIRA?


was (Author: sjlee0):
The latest patch looks good to me. The unit test that timed out is known to 
happen occasionally (can't locate an open JIRA for that one though). The 
deprecation warning is also an unfortunately result of {{DistributedCache}} 
being deprecated.

I do have a small quibble with the title of the JIRA. The title may give you an 
impression that with this JIRA one could use a wildcard in specifying the 
libjars *argument*. But that's not the case. What this JIRA does (in 
conjunction with YARN-4958) is to collapse the libjar directory in the 
configuration as well as the container launch context. But the current title 
can be confused with HADOOP-12747 ("support wildcard in libjars argument"). Can 
we modify the title to better describe this JIRA?

> -libjars should use wildcards to reduce the application footprint in the 
> state store
> 
>
> Key: MAPREDUCE-6719
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: MAPREDUCE-6719.001.patch, MAPREDUCE-6719.002.patch
>
>
> When using the -libjars option to add classes to the classpath, every library 
> so added is explicitly listed in the ContainerLaunchContext's local resources 
> even though they're all uploaded to the same directory in HDFS. When using 
> tools like Crunch without an uber JAR or when trying to take advantage of the 
> shared cache, the number of libraries can be quite large. We've seen many 
> cases where we had to turn down the max number of applications to prevent ZK 
> from running out of heap because of the size of the state store entries.
> This JIRA proposes to allow for wildcards both in the internal processing of 
> the -libjars switch and in paths added through the Job and DistributedCache 
> classes. Rather than listing all files independently, this JIRA proposes to 
> replace the complete list of libdir files with the wildcarded libdir 
> directory, e.g. "libdir/*". This behavior is the same as the current behavior 
> when using -libjars, but avoids explicitly listing every file.
> This capability will also be exposed by the 
> {{DistributedCache.addCacheFile()}} method.
> See YARN-4958 for the NM side of the implementation and additional discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6719) -libjars should use wildcards to reduce the application footprint in the state store

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340813#comment-15340813
 ] 

Sangjin Lee commented on MAPREDUCE-6719:


The latest patch looks good to me. The unit test that timed out is known to 
happen occasionally (can't locate an open JIRA for that one though). The 
deprecation warning is also an unfortunately result of {{DistributedCache}} 
being deprecated.

I do have a small quibble with the title of the JIRA. The title may give you an 
impression that with this JIRA one could use a wildcard in specifying the 
libjars *argument*. But that's not the case. What this JIRA does (in 
conjunction with YARN-4958) is to collapse the libjar directory in the 
configuration as well as the container launch context. But the current title 
can be confused with HADOOP-12747 ("support wildcard in libjars argument"). Can 
we modify the title to better describe this JIRA?

> -libjars should use wildcards to reduce the application footprint in the 
> state store
> 
>
> Key: MAPREDUCE-6719
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: MAPREDUCE-6719.001.patch, MAPREDUCE-6719.002.patch
>
>
> When using the -libjars option to add classes to the classpath, every library 
> so added is explicitly listed in the ContainerLaunchContext's local resources 
> even though they're all uploaded to the same directory in HDFS. When using 
> tools like Crunch without an uber JAR or when trying to take advantage of the 
> shared cache, the number of libraries can be quite large. We've seen many 
> cases where we had to turn down the max number of applications to prevent ZK 
> from running out of heap because of the size of the state store entries.
> This JIRA proposes to allow for wildcards both in the internal processing of 
> the -libjars switch and in paths added through the Job and DistributedCache 
> classes. Rather than listing all files independently, this JIRA proposes to 
> replace the complete list of libdir files with the wildcarded libdir 
> directory, e.g. "libdir/*". This behavior is the same as the current behavior 
> when using -libjars, but avoids explicitly listing every file.
> This capability will also be exposed by the 
> {{DistributedCache.addCacheFile()}} method.
> See YARN-4958 for the NM side of the implementation and additional discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6719) -libjars should use wildcards to reduce the application footprint in the state store

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339939#comment-15339939
 ] 

Sangjin Lee commented on MAPREDUCE-6719:


{noformat}
diff --git 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/nbproject/project.properties
 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/nbproject/project.properties
new file mode 100644
index 000..e69de29
{noformat}
Is the above file supposed to be part of this patch? It seems to me that either 
the file's content should be in the patch or it should be removed from the 
patch.


> -libjars should use wildcards to reduce the application footprint in the 
> state store
> 
>
> Key: MAPREDUCE-6719
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: MAPREDUCE-6719.001.patch
>
>
> When using the -libjars option to add classes to the classpath, every library 
> so added is explicitly listed in the ContainerLaunchContext's local resources 
> even though they're all uploaded to the same directory in HDFS. When using 
> tools like Crunch without an uber JAR or when trying to take advantage of the 
> shared cache, the number of libraries can be quite large. We've seen many 
> cases where we had to turn down the max number of applications to prevent ZK 
> from running out of heap because of the size of the state store entries.
> This JIRA proposes to allow for wildcards both in the internal processing of 
> the -libjars switch and in paths added through the Job and DistributedCache 
> classes. Rather than listing all files independently, this JIRA proposes to 
> replace the complete list of libdir files with the wildcarded libdir 
> directory, e.g. "libdir/*". This behavior is the same as the current behavior 
> when using -libjars, but avoids explicitly listing every file.
> This capability will also be exposed by the 
> {{DistributedCache.addCacheFile()}} method.
> See YARN-4958 for the NM side of the implementation and additional discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6688) Store job configurations in Timeline Service v2

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6688:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: YARN-2928
   Status: Resolved  (was: Patch Available)

Committed the patch (04). Thanks [~varun_saxena] for your contribution, and 
[~gtCarrera9], [~Naganarasimha], and  [~vrushalic] for your reviews!

> Store job configurations in Timeline Service v2
> ---
>
> Key: MAPREDUCE-6688
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Junping Du
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Fix For: YARN-2928
>
> Attachments: MAPREDUCE-6688-YARN-2928.01.patch, 
> MAPREDUCE-6688-YARN-2928.02.patch, MAPREDUCE-6688-YARN-2928.03.patch, 
> MAPREDUCE-6688-YARN-2928.04.patch, MAPREDUCE-6688-YARN-2928.v2.01.patch, 
> MAPREDUCE-6688-YARN-2928.v2.02.patch, YARN-3959-YARN-2928.01.patch
>
>
> We already have configuration field in HBase schema for application entity. 
> We need to make sure AM write it out when it get launched.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6688) Store job configurations in Timeline Service v2

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6688:
---
Attachment: MAPREDUCE-6688-YARN-2928.04.patch

Removed several unused imports for the checkstyle issues.

> Store job configurations in Timeline Service v2
> ---
>
> Key: MAPREDUCE-6688
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Junping Du
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6688-YARN-2928.01.patch, 
> MAPREDUCE-6688-YARN-2928.02.patch, MAPREDUCE-6688-YARN-2928.03.patch, 
> MAPREDUCE-6688-YARN-2928.04.patch, MAPREDUCE-6688-YARN-2928.v2.01.patch, 
> MAPREDUCE-6688-YARN-2928.v2.02.patch, YARN-3959-YARN-2928.01.patch
>
>
> We already have configuration field in HBase schema for application entity. 
> We need to make sure AM write it out when it get launched.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268123#comment-15268123
 ] 

Sangjin Lee commented on MAPREDUCE-6688:


The latest patch (v2.02) looks good. I'll wait for jenkins to come back.

> Store job configurations in Timeline Service v2
> ---
>
> Key: MAPREDUCE-6688
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Junping Du
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6688-YARN-2928.01.patch, 
> MAPREDUCE-6688-YARN-2928.02.patch, MAPREDUCE-6688-YARN-2928.03.patch, 
> MAPREDUCE-6688-YARN-2928.v2.01.patch, MAPREDUCE-6688-YARN-2928.v2.02.patch, 
> YARN-3959-YARN-2928.01.patch
>
>
> We already have configuration field in HBase schema for application entity. 
> We need to make sure AM write it out when it get launched.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15267638#comment-15267638
 ] 

Sangjin Lee commented on MAPREDUCE-6688:


Agreeing with [~gtCarrera9], I'd like to propose that we proceed with the 
version without refactoring for now. What do others think?

I took a look at that patch (v.2.01), and had a few comments.

(JobHistoryEventHandler.java)
- l.1129-1130: The id and type are set twice (once in {{createJobEntity()}}, 
another here). I think these second set calls should be removed.
- l.1140-1145: if a single key/value exceeds the limit, it will do a spurious 
{{putEntities()}} call with empty entities unnecessarily; I think there should 
be an additional check to see if there is anything to send

Did we reach a conclusion on whether these should be {{putEntities()}} 
(blocking) or {{putEntitiesAsync()}}? I'm leaning slightly towards the blocking 
call, as the config is arguably more important information than periodic metric 
updates for example. I don't think we have established a firm rule on what 
should be sync and async, however, and we might need to revisit it as a whole 
later.

> Store job configurations in Timeline Service v2
> ---
>
> Key: MAPREDUCE-6688
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Junping Du
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6688-YARN-2928.01.patch, 
> MAPREDUCE-6688-YARN-2928.02.patch, MAPREDUCE-6688-YARN-2928.03.patch, 
> MAPREDUCE-6688-YARN-2928.v2.01.patch, YARN-3959-YARN-2928.01.patch
>
>
> We already have configuration field in HBase schema for application entity. 
> We need to make sure AM write it out when it get launched.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265642#comment-15265642
 ] 

Sangjin Lee commented on MAPREDUCE-6688:


If we don't have enough time to resolve these points, I think we have options 
to do the simple thing (emit the config in a single call and defer the size 
consideration in a later JIRA) or even defer this JIRA itself.

> Store job configurations in Timeline Service v2
> ---
>
> Key: MAPREDUCE-6688
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Reporter: Junping Du
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6688-YARN-2928.01.patch, 
> YARN-3959-YARN-2928.01.patch
>
>
> We already have configuration field in HBase schema for application entity. 
> We need to make sure AM write it out when it get launched.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265640#comment-15265640
 ] 

Sangjin Lee commented on MAPREDUCE-6424:


Thanks [~Naganarasimha]! I'm +1 on the latest patch. I'd like to give a little 
time for [~gtCarrera9] and [~varun_saxena]. Could you guys review it, and 
commit it if you are +1 too?

> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, 
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, 
> MAPREDUCE-6424-YARN-2928.v1.004.patch, MAPREDUCE-6424-YARN-2928.v1.005.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265362#comment-15265362
 ] 

Sangjin Lee commented on MAPREDUCE-6424:


Thanks for tracking down the unit test failure until late/early. The checkstyle 
issues look trivial to correct. [~Naganarasimha], could I bother you to fix 
them? Then I'd be +1 on the patch. What do other think?

> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, 
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, 
> MAPREDUCE-6424-YARN-2928.v1.004.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265194#comment-15265194
 ] 

Sangjin Lee commented on MAPREDUCE-6688:


I am comfortable with a constant (not much users would gain by tweaking this 
value high or low), but I'd like to hear what others think...

> Store job configurations in Timeline Service v2
> ---
>
> Key: MAPREDUCE-6688
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Reporter: Junping Du
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-3959-YARN-2928.01.patch
>
>
> We already have configuration field in HBase schema for application entity. 
> We need to make sure AM write it out when it get launched.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6688) Store job configurations in Timeline Service v2

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264842#comment-15264842
 ] 

Sangjin Lee commented on MAPREDUCE-6688:


bq. If we have done something in hRaven, we can probably adopt the same 
approach if conflicting opinions do not exist.
cc [~vrushalic] and [~jrottinghuis]

bq. Maybe we can go ahead with splitting with a fixed static constant for size 
of configs to be sent in one shot ? Is 100 kb fine ? Thoughts ?
I'm +1 on doing multiple {{putEntities()}} calls with the upper bound of 100 KB 
with one exception; if a single value exceeds that size. We just need to decide 
on a behavior, and I would lean towards still sending it for now if a single 
key/value exceeds the upper bound. Would it be feasible?

> Store job configurations in Timeline Service v2
> ---
>
> Key: MAPREDUCE-6688
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Reporter: Junping Du
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-3959-YARN-2928.01.patch
>
>
> We already have configuration field in HBase schema for application entity. 
> We need to make sure AM write it out when it get launched.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264714#comment-15264714
 ] 

Sangjin Lee commented on MAPREDUCE-6424:


I'm fine with the suggestion.

Be reminded that the timeline client combines multiple (async) writes to arrive 
at a reasonable level of writes to the timeline collector. So in general, the 
penalty for making fine-grained putEntitiesAsync() calls shouldn't be too big.

> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, 
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264686#comment-15264686
 ] 

Sangjin Lee commented on MAPREDUCE-6424:


Yeah that might be a better fit. In general, any place where we have a need to 
loop to wait until a test condition is satisfied is a candidate to consider 
{{GenericTestUtils.waitFor()}}.

> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, 
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263116#comment-15263116
 ] 

Sangjin Lee commented on MAPREDUCE-6424:


(JobHistoryEventHandler.java)
- l.1053: I believe this causes a checkstyle violation (line too long)
- l.1054: {{timestamp}} here is no longer used, and should be removed

(JobFinishedEvent.java)
- l.161: Sorry I should have been clearer, but since we have a clear timestamp 
associated with this event ({{finishTime}}), I think we ought to use that time 
over the current time millis. Thoughts? Same for {{MapAttemptFinishedEvent}}, 
{{ReduceAttemptFinishedEvent}}, {{TaskAttemptFinishedEvent}}, 
{{TaskAttemptUnsuccessfulCompletionEvent}}, {{TaskFailedEvent}}, and 
{{TaskFinishedEvent}}

(TestMRTimelineEventHandling.java)
- l.337-359: Is it possible to use {{GenericTestUtils.waitFor()}} for this?


> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, 
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262545#comment-15262545
 ] 

Sangjin Lee commented on MAPREDUCE-6424:


Please ignore my comment about the findbugs warnings. They appear unrelated to 
our branch.

> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, 
> MAPREDUCE-6424-YARN-2928.v1.002.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262537#comment-15262537
 ] 

Sangjin Lee commented on MAPREDUCE-6424:


Thanks [~Naganarasimha] for the updated patch! It looks good for the most part. 
Some feedback:

(JobHistoryEventHandler.java)
- l.1055-1067: it might be better if we do this only if the metrics is not null 
(i.e. moving the {{if (timelineMetrics != null)}}) check to the outside
- l.1057, 1061: I don't think this event is needed at the application entity 
(all we need are the metrics)
- l.1231: I think this is a debatable point, but since these are final metric 
values would it be better to write them via {{putEntities()}}?

(JobHistoryEventUtils.java)
- l.68: This might be a small point, but how about adding {{long timestamp}} as 
an input argument so that all metrics can have the same timestamp? That way we 
can precisely set the timestamp if needed.

(TestMRTimelineEventHandling.java)
- can we add a test to check the metrics for the application entity?

It looks like we have a couple of pre-existing findbugs warnings. Did you have 
a chance to see if they are straightforward to fix? Also, the checkstyle 
violations with the patch should be easy to fix. Are the unit test failures 
unrelated?



> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, 
> MAPREDUCE-6424-YARN-2928.v1.002.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259517#comment-15259517
 ] 

Sangjin Lee commented on MAPREDUCE-6424:


{quote}
This seems to be a good idea for oozie like flows but just one query, would it 
be good for app to publish metrics in SystemEntities ? IIRC YARN-3401 was 
raised to block it later on ? so do we need to support some other mechanism to 
publish only metrics ?
{quote}

We may need to refine what we do on YARN-3401. YARN-3401 notwithstanding, we 
still have a need to support aggregating framework-owned metrics. Consider our 
famous example: HDFS bytes written for the job. If we emit it for the MR job 
entity, then it would not benefit from aggregation onto the flow and so on. One 
can query for the job entities and get the HDFS bytes written for individual 
jobs, but there would be no flow aggregation. By adding the metrics on the YARN 
application entity, the flow aggregation will kick in naturally and be able to 
show it at the flow run level.

If we don't do this, then we need to find a way to transfer metrics from 
certain framework-specific entities to the application entities so that they 
can be aggregated. I would think that would be considerably more involved.

{quote}
Ok basically was trying to understand for the next jira, may be we can discuss 
more there. Query was : Apart from metrics getting published at the end of each 
task(M/R) we need to publish metrics when task(M/R) is running or have some way 
to share running task's (M/R) metrics to AM(via umbilical proto) and AM publish 
aggregated running tasks (M/R) metrics periodically ?
{quote}
I see. Yeah, I haven't given a lot of thoughts on exactly how we might do this. 
As you mentioned, we can carry on the discussion in a new JIRA.

> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259240#comment-15259240
 ] 

Sangjin Lee commented on MAPREDUCE-6424:


Thanks [~Naganarasimha] for your patch!

I think we said it is OK to emit them as is and capture the work of emitting 
them periodically in a later JIRA.

One thing to add, though: I mentioned it elsewhere but we also need to emit the 
metrics at the *YARN application* level if we want them aggregated to the 
flows. Therefore, we should add code that emits the metrics for the YARN 
application entity in addition to the MR job entity. I'd say that's actually 
even more important than metrics at the job entity. The metrics at the 
MR-specific entities are there to view, but they will not be aggregated. On the 
other hand, the YARN application entity is the focal point of real time flow 
aggregation.

> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6315) Implement retrieval of logs for crashed MR-AM via jhist in the staging directory

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258663#comment-15258663
 ] 

Sangjin Lee commented on MAPREDUCE-6315:


Thanks [~jira.shegalov]! This is a much-needed enhancement. I agree with most 
of Ming's comments. Especially it would be awesome if the web side can benefit 
from this.

> Implement retrieval of logs for crashed MR-AM via jhist in the staging 
> directory
> 
>
> Key: MAPREDUCE-6315
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6315
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client, mr-am
>Affects Versions: 2.7.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Critical
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-6315.001.patch, MAPREDUCE-6315.002.patch, 
> MAPREDUCE-6315.003.patch
>
>
> When all AM attempts crash, there is no record of them in JHS. Thus no easy 
> way to get the logs. This JIRA automates the procedure by utilizing the jhist 
> file in the staging directory. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5817) Mappers get rescheduled on node transition even after all reducers are completed

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246018#comment-15246018
 ] 

Sangjin Lee commented on MAPREDUCE-5817:


I have no objections to backporting this to branch-2.7 or branch-2.6. Thanks!

> Mappers get rescheduled on node transition even after all reducers are 
> completed
> 
>
> Key: MAPREDUCE-5817
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5817
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.3.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-5817.001.patch, MAPREDUCE-5817.002.patch, 
> mapreduce-5817.patch
>
>
> We're seeing a behavior where a job runs long after all reducers were already 
> finished. We found that the job was rescheduling and running a number of 
> mappers beyond the point of reducer completion. In one situation, the job ran 
> for some 9 more hours after all reducers completed!
> This happens because whenever a node transition (to an unusable state) comes 
> into the app master, it just reschedules all mappers that already ran on the 
> node in all cases.
> Therefore, if any node transition has a potential to extend the job period. 
> Once this window opens, another node transition can prolong it, and this can 
> happen indefinitely in theory.
> If there is some instability in the pool (unhealthy, etc.) for a duration, 
> then any big job is severely vulnerable to this problem.
> If all reducers have been completed, JobImpl.actOnUnusableNode() should not 
> reschedule mapper tasks. If all reducers are completed, the mapper outputs 
> are no longer needed, and there is no need to reschedule mapper tasks as they 
> would not be consumed anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee resolved MAPREDUCE-6372.

   Resolution: Resolved
Fix Version/s: YARN-2928

I have addressed the known issues as part of MAPREDUCE-6546.

> clean up several issues with TimelineServicePerformance
> ---
>
> Key: MAPREDUCE-6372
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>  Labels: yarn-2928-1st-milestone
> Fix For: YARN-2928
>
>
> We found a few issues with the TimelineServicePerformanceV2 test driver while 
> running it for the performance tests. Filing this JIRA to fix those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187531#comment-15187531
 ] 

Sangjin Lee commented on MAPREDUCE-6546:


Thanks [~Naganarasimha] for your review and commit!

> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
> Fix For: YARN-2928
>
> Attachments: MAPREDUCE-6546-YARN-2928.01.patch, 
> MAPREDUCE-6546-YARN-2928.02.patch, MAPREDUCE-6546-YARN-2928.03.patch
>
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184358#comment-15184358
 ] 

Sangjin Lee commented on MAPREDUCE-6546:


Oops sorry about that. I should have printed out the message before cutting the 
patch. Yes, please feel free to add the closing parenthesis. Thanks!

> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6546-YARN-2928.01.patch, 
> MAPREDUCE-6546-YARN-2928.02.patch, MAPREDUCE-6546-YARN-2928.03.patch
>
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6546:
---
Attachment: MAPREDUCE-6546-YARN-2928.03.patch

Posted patch v.3.

Addressed Naga's feedback. Thanks for the review!

> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6546-YARN-2928.01.patch, 
> MAPREDUCE-6546-YARN-2928.02.patch, MAPREDUCE-6546-YARN-2928.03.patch
>
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6546:
---
Attachment: MAPREDUCE-6546-YARN-2928.02.patch

Posted patch v.2.

> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6546-YARN-2928.01.patch, 
> MAPREDUCE-6546-YARN-2928.02.patch
>
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157507#comment-15157507
 ] 

Sangjin Lee commented on MAPREDUCE-6546:


Thanks [~garlanaganarasi...@huawei.com] for your review. Those are good points. 
I'll post an updated patch shortly.

> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6546-YARN-2928.01.patch
>
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149805#comment-15149805
 ] 

Sangjin Lee commented on MAPREDUCE-6546:


The unit test issues are unrelated.

> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6546-YARN-2928.01.patch
>
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6546:
---
Attachment: MAPREDUCE-6546-YARN-2928.01.patch

Posted patch v.1.

The test main class is now merged into {{TimelineServicePerformance}}. Also 
moved all v2-specific classes to the same package (o.a.h.mapreduce). The diff 
is somewhat hard to read because some classes and code were moved. Let me know 
if you want me to add more details to aid your review.

> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6546-YARN-2928.01.patch
>
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6546:
---
Status: Patch Available  (was: In Progress)

> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6546-YARN-2928.01.patch
>
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work stopped] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on MAPREDUCE-6372 stopped by Sangjin Lee.
--
> clean up several issues with TimelineServicePerformance
> ---
>
> Key: MAPREDUCE-6372
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>  Labels: yarn-2928-1st-milestone
>
> We found a few issues with the TimelineServicePerformanceV2 test driver while 
> running it for the performance tests. Filing this JIRA to fix those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6546:
---
Labels: yarn-2928-1st-milestone  (was: )

> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on MAPREDUCE-6546 started by Sangjin Lee.
--
> reconcile the two versions of the timeline service performance tests
> 
>
> Key: MAPREDUCE-6546
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
>
> The trunk now has a version of the timeline service performance test 
> (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance 
> test, and these two versions are quite similar (by design).
> We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6577:
---
Fix Version/s: 2.8.0

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4
>
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085803#comment-15085803
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


Thanks for pointing that out. I somehow (mistakenly) thought that 2.8.0 was 
already being cut. I just picked the fix to 2.8.0.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4
>
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083856#comment-15083856
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


Thanks [~cnauroth]! [~jlowe], do let me know if you're +1 on the latest patch. 
I can commit it then.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6577:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.6.4
   2.9.0
   2.7.3
   Status: Resolved  (was: Patch Available)

Committed the patch to trunk, branch-2, branch-2.7, and branch-2.6. Thanks 
[~jlowe] and [~cnauroth] for your reviews and suggestions!

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Fix For: 2.7.3, 2.9.0, 2.6.4
>
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6588) default values for PATH for tasks are not quite correct on Windows

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6588:
---
Summary: default values for PATH for tasks are not quite correct on Windows 
 (was: default values for PATH for tasks are not quite correct)

> default values for PATH for tasks are not quite correct on Windows
> --
>
> Key: MAPREDUCE-6588
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6588
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>
> There are several issues with the way PATH are handled for MR tasks and AMs 
> on Windows.
> - it does not get the PWD as part of the path, unlike the non-Windows case 
> (see MAPREDUCE-6021)
> - the MR AM doesn't get the Hadoop's bin directory in the path explicitly 
> through mapred configuration; rather, it implicitly inherits it from the node 
> manager (see MAPREDUCE-6577)
> - the default value for the MR task PATH uses "%...%" macros which may be 
> expanded on the client side prematurely; this is probably OK for tasks (as 
> it's done by the AM), but not OK for AMs; the late-binding cross-platform 
> macros should be used
> I took a stab at addressing these as part of MAPREDUCE-6577, but I realized 
> that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus, 
> I'm filing a separate JIRA to address the Windows side of things.
> On a related note, there are quite a few brittle tests on testing these 
> (TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such 
> a way that the tests pass against the current behavior but don't provide a 
> lot of robust test value. We should also update those tests to make them more 
> robust.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6588) default values for PATH for tasks are not quite correct

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6588:
--

 Summary: default values for PATH for tasks are not quite correct
 Key: MAPREDUCE-6588
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6588
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.6.0
Reporter: Sangjin Lee


There are several issues with the way PATH are handled for MR tasks and AMs on 
Windows.

- it does not get the PWD as part of the path, unlike the non-Windows case (see 
MAPREDUCE-6021)
- the MR AM doesn't get the Hadoop's bin directory in the path explicitly 
through mapred configuration; rather, it implicitly inherits it from the node 
manager (see MAPREDUCE-6577)
- the default value for the MR task PATH uses "%...%" macros which may be 
expanded on the client side prematurely; this is probably OK for tasks (as it's 
done by the AM), but not OK for AMs; the late-binding cross-platform macros 
should be used

I took a stab at addressing these as part of MAPREDUCE-6577, but I realized 
that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus, 
I'm filing a separate JIRA to address the Windows side of things.

On a related note, there are quite a few brittle tests on testing these 
(TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such a 
way that the tests pass against the current behavior but don't provide a lot of 
robust test value. We should also update those tests to make them more robust.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6577:
---
Attachment: MAPREDUCE-6577.04.patch

Posted patch v.4.

This keeps the Windows behavior identical to the current trunk. As a result, 
some of the code changes are slightly uglier. :)

Another interesting change: as part of testing this pretty extensively, I 
stumbled onto the fact that HADOOP_COMMON_HOME was not being set in the hadoop 
unit tests. There are a lot of places where HADOOP_COMMON_HOME is used 
(including this one), and I suppose this problem was masked because most paths 
were being inherited from the node manager (mini-YARN cluster).

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6588) default values for PATH for tasks are not quite correct

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070365#comment-15070365
 ] 

Sangjin Lee commented on MAPREDUCE-6588:


One interesting observation I made while trying to use the cross-platform 
macros for the MR AM as a test:

{noformat}
PATH={{PATH}};{{HADOOP_COMMON_HOME}}\bin
{noformat}

The node manager sets the following line in the batch script that launches the 
AM:

{noformat}
@set PATH=%PATH%;%HADOOP_COMMON_HOME%\bin
{noformat}

However, on the AM process, the PATH environment variable is as follows:

{noformat}
PATH=%PATH%;%HADOOP_COMMON_HOME%\bin;C:\hadoop\hadoop-mapreduce-project\hadoop-mapreduce-client\hadoop-mapreduce-client-jobclient/../../../hadoop-common-project/hadoop-common/target\bin
{noformat}

While the substitution for HADOOP_COMMON_HOME is made, PATH is not. And also 
the original macro is preserved instead of being replaced. Maybe this is 
because I'm not familiar with Windows batch scripting, but this causes the 
existing path value to be dropped.

> default values for PATH for tasks are not quite correct
> ---
>
> Key: MAPREDUCE-6588
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6588
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>
> There are several issues with the way PATH are handled for MR tasks and AMs 
> on Windows.
> - it does not get the PWD as part of the path, unlike the non-Windows case 
> (see MAPREDUCE-6021)
> - the MR AM doesn't get the Hadoop's bin directory in the path explicitly 
> through mapred configuration; rather, it implicitly inherits it from the node 
> manager (see MAPREDUCE-6577)
> - the default value for the MR task PATH uses "%...%" macros which may be 
> expanded on the client side prematurely; this is probably OK for tasks (as 
> it's done by the AM), but not OK for AMs; the late-binding cross-platform 
> macros should be used
> I took a stab at addressing these as part of MAPREDUCE-6577, but I realized 
> that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus, 
> I'm filing a separate JIRA to address the Windows side of things.
> On a related note, there are quite a few brittle tests on testing these 
> (TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such 
> a way that the tests pass against the current behavior but don't provide a 
> lot of robust test value. We should also update those tests to make them more 
> robust.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070370#comment-15070370
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


After looking into the Windows side of things for a while, I realize that there 
are bigger issues to address on that side that go beyond the scope of this 
JIRA. Also, Windows isn't exactly broken in the same manner as Linux is by 
MAPREDUCE-6021 as it didn't really touch Windows.

I filed a separate JIRA to address the Windows parity and less major issues 
(MAPREDUCE-6588). I propose we tackle the Windows aspect in that JIRA.

I'll update the patch shortly to preserve the Windows behavior as it is today.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068539#comment-15068539
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


Thanks Chris. I actually stated down my path on setting that up, but I was 
having tons of trouble setting up the Visual Studio (or Windows SDK?). Even 
with the native build turned off, I kept getting errors around msbuild and 
Visual Studio. I tried all kinds of different versions (10, 2013, 2015) with no 
success. I'll give it one more shot.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066942#comment-15066942
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


Thanks [~laurentgo] and [~cnauroth] for the pointers! Yes, I am aware that Mac 
uses DYLD_LIBRARY_PATH, but there seem to be more quirks (perhaps 
version-dependent?) that mess with that. Since it's an existing larger issue, I 
think it's safe to say it's outside the scope of this JIRA.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066815#comment-15066815
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


The latest patch (v.3) should now be good to review.

I was thrown off for a bit because Mac OS (on which I was testing some of these 
unit tests) doesn't seem to handle LD_LIBRARY_PATH in the same way as Linux. In 
fact, the trunk version of TestMiniMRChildTask does not pass on Mac. But on 
Linux, the test is fine before and after the patch.

I would appreciate your review. Thanks!

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067641#comment-15067641
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


No worries. Thanks for checking. It's rather difficult to acquire a Windows 
machine that's ready to build hadoop.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6577:
---
Attachment: MAPREDUCE-6577.03.patch

Posted patch v.3.

Fixed TestMiniMRChildTask#testMapRedExecutionEnv(). Removed the first part of 
the test as it is not meaningful to test the HADOOP_COMMON_HOME substitution as 
it is no longer substituted with the late binding.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, 
> MAPREDUCE-6577.03.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064181#comment-15064181
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


Thanks for the pointer [~jlowe]! MRApps.crossPlatformifyMREnv() does that only 
if mapreduce.app-submission.cross-platform is true (which is false by default)? 
So it looks like it'd be safer if I used \{\{ and \}\} directly, no? Also, the 
method requires a Configuration object and an Environment object, so I'm not 
sure if that works in MRJobConfig.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064196#comment-15064196
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


Thanks for the clarification. Yes, that's what I'm going to do now that the 
variables are not going to be expanded.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6577:
---
Attachment: MAPREDUCE-6577.02.patch

Posted patch v.2 that addresses Jason's review comments.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064950#comment-15064950
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


It looks like TestMiniMRChildTask#testMapRedExecutionEnv() needs updating. 
Please hold off on committing it until I update the patch. Thanks.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6577:
---
Status: Patch Available  (was: Open)

Posted patch v.1.

Added the default for the AM admin user env (which has the same value as the 
non-AM admin user env).

Also did a small amount of yak-shaving. I suspect that the lib path handling on 
Windows is not quite right (PWD was not being added for Windows). I tried to 
make it a little more cross-platform there.

Folks who use Windows ([~cnauroth]?), I'd greatly appreciate it if you could 
review the patch from that perspective.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6577:
---
Attachment: MAPREDUCE-6577.01.patch

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063442#comment-15063442
 ] 

Sangjin Lee commented on MAPREDUCE-6577:


All the issues that are flagged by jenkins (findbugs, javac, checkstyle, and 
unit tests) appear to be existing issues that are unrelated to the patch.

I'd appreciate your review on the patch. Thanks!

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> 
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will 
> fail. For example, an uber-AM with lz4 compression for the mapper task will 
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
> available
>   at 
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>   at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
>   at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
>   at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6577:
--

 Summary: MR AM unable to load native library without 
MR_AM_ADMIN_USER_ENV set
 Key: MAPREDUCE-6577
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.6.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Priority: Critical


If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:

{noformat}
2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... using builtin-java 
classes where applicable
{noformat}

As a result, any code that needs the hadoop native library in the MR AM will 
fail. For example, an uber-AM with lz4 compression for the mapper task will 
fail:
{noformat}
2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
(uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
available
at 
org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
at 
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at 
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)