[jira] [Commented] (MAPREDUCE-6875) mapred-site.xml in bin tarball lacks a license
[ https://issues.apache.org/jira/browse/MAPREDUCE-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958285#comment-15958285 ] Haibo Chen commented on MAPREDUCE-6875: --- There is a etc/hadoop/mapred-site.xml.template file that users can start from. Do you think that is sufficient enough? > mapred-site.xml in bin tarball lacks a license > -- > > Key: MAPREDUCE-6875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6875 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Priority: Blocker > > mapred-site.xml file needs a license. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6875) mapred-site.xml in bin tarball lacks a license
[ https://issues.apache.org/jira/browse/MAPREDUCE-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958258#comment-15958258 ] Allen Wittenauer commented on MAPREDUCE-6875: - Oh wait a sec. I see what is going on. mapred-site.xml doesn't actually exist but others do. My auto-populate stuff is creating it. So the real bug here is that mapred-site.xml doesn't exist. > mapred-site.xml in bin tarball lacks a license > -- > > Key: MAPREDUCE-6875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6875 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Priority: Blocker > > mapred-site.xml file needs a license. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6875) mapred-site.xml in bin tarball lacks a license
[ https://issues.apache.org/jira/browse/MAPREDUCE-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958255#comment-15958255 ] Allen Wittenauer commented on MAPREDUCE-6875: - etc/hadoop/mapred-site.xml > mapred-site.xml in bin tarball lacks a license > -- > > Key: MAPREDUCE-6875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6875 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Priority: Blocker > > mapred-site.xml file needs a license. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6875) mapred-site.xml in bin tarball lacks a license
[ https://issues.apache.org/jira/browse/MAPREDUCE-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958242#comment-15958242 ] Haibo Chen commented on MAPREDUCE-6875: --- [~aw] I cannot find a file named mapred-site.xml in the tarball. Do you mind sharing the relative path? > mapred-site.xml in bin tarball lacks a license > -- > > Key: MAPREDUCE-6875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6875 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Priority: Blocker > > mapred-site.xml file needs a license. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958114#comment-15958114
]
Hudson commented on MAPREDUCE-6846:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11537 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11537/])
MAPREDUCE-6846. Fragments specified for libjar paths are not handled (templedf:
rev fc0885da294490c3984c2231a4d35f89b3b520d4)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobResourceUploader.java
> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: MAPREDUCE-6846-trunk.001.patch,
> MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch,
> MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch,
> MAPREDUCE-6846-trunk.006.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser,
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute
> path (i.e. /home/mapred
[jira] [Updated] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6846:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.9.0
Status: Resolved (was: Patch Available)
Thanks for the patch, [~ctrezzo], and thanks for the reviews, [~sjlee0] and
[~jlowe]! Committed to trunk and branch-2.
> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: MAPREDUCE-6846-trunk.001.patch,
> MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch,
> MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch,
> MAPREDUCE-6846-trunk.006.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser,
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute
> path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e.
> test.txt#frag.txt) ||
> || -libjars | FileNotFound | FileNotFound|FileNotFound|
> || -files | (/) | (/) | (/) |
> || -archives | (/) | (/) | (/) |
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
T
[jira] [Updated] (MAPREDUCE-6875) mapred-site.xml in bin tarball lacks a license
[ https://issues.apache.org/jira/browse/MAPREDUCE-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6875: Affects Version/s: 3.0.0-alpha3 > mapred-site.xml in bin tarball lacks a license > -- > > Key: MAPREDUCE-6875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6875 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Priority: Blocker > > mapred-site.xml file needs a license. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6875) mapred-site.xml in bin tarball lacks a license
Allen Wittenauer created MAPREDUCE-6875: --- Summary: mapred-site.xml in bin tarball lacks a license Key: MAPREDUCE-6875 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6875 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Allen Wittenauer Priority: Blocker mapred-site.xml file needs a license. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957590#comment-15957590
]
Sangjin Lee commented on MAPREDUCE-6846:
Thanks [~dan...@cloudera.com]! Would you do the honors of committing this?
> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch,
> MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch,
> MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch,
> MAPREDUCE-6846-trunk.006.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser,
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute
> path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e.
> test.txt#frag.txt) ||
> || -libjars | FileNotFound | FileNotFound|FileNotFound|
> || -files | (/) | (/) | (/) |
> || -archives | (/) | (/) | (/) |
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957499#comment-15957499
]
Haibo Chen commented on MAPREDUCE-6871:
---
Thanks [~rkanter] for the patch! A few comments
1) Do you think having regular expression for the configuration is more robust?
rack1/node2 will be treated as a node name with current implementation. If this
will cause application failure later, better we catch it early before
submission.
2) Maybe rename mapreduce.job.am.resource-request.strict.locality to
mapreduce.job.am.resource-request-strict-locality ?
3) The different test cases can be broken into different test methods
individually, so that one test case failure won't disguise failures in
following test cases.
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957313#comment-15957313
]
Daniel Templeton commented on MAPREDUCE-6846:
-
Latest changes look good to me. I'd say that you should make sure to update
the docs to reflect the new behavior, but there aren't any! ;)
> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch,
> MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch,
> MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch,
> MAPREDUCE-6846-trunk.006.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser,
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute
> path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e.
> test.txt#frag.txt) ||
> || -libjars | FileNotFound | FileNotFound|FileNotFound|
> || -files | (/) | (/) | (/) |
> || -archives | (/) | (/) | (/) |
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoo
[jira] [Commented] (MAPREDUCE-6824) TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
[
https://issues.apache.org/jira/browse/MAPREDUCE-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957254#comment-15957254
]
Chris Trezzo commented on MAPREDUCE-6824:
-
Thanks [~ajisakaa]!
> TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
> ---
>
> Key: MAPREDUCE-6824
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6824
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Trivial
> Labels: newbie
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: MAPREDUCE-6824-branch-2.001.patch,
> MAPREDUCE-6824-trunk.001.patch, MAPREDUCE-6824-trunk.002.patch,
> MAPREDUCE-6824-trunk.003.patch
>
>
> bq.
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752:
> private static ContainerLaunchContext createCommonContainerLaunchContext(:3:
> Method length is 172 lines (max allowed is 150).
> {{TaskAttemptImpl#createCommonContainerLaunchContext}} is longer than 150
> lines and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6867) ApplicationMaster hung on OOM Error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956975#comment-15956975
]
Hadoop QA commented on MAPREDUCE-6867:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 0s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 10s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 42s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 41s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862075/MAPREDUCE-6867.patch |
| JIRA Issue | MAPREDUCE-6867 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ceaa3bffc356 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 34ab8e7 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6948/artifact/patchprocess/whitespace-eol.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6948/artifact/patchprocess/whitespace-tabs.txt
|
[jira] [Updated] (MAPREDUCE-6867) ApplicationMaster hung on OOM Error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6867:
Status: Patch Available (was: Open)
> ApplicationMaster hung on OOM Error
> ---
>
> Key: MAPREDUCE-6867
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6867
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Bilwa S T
>Assignee: Bilwa S T
> Attachments: MAPREDUCE-6867.patch
>
>
> Whenever OOM Error is thown, {{YarnUncaughtExceptionHandler}} will call
> {{ExitUtil.halt(-1)}}.But while halting, OOM might occur which is not
> handled.
> We came across a scenario where in when we submit mapreduce application ,OOM
> error occured in {{committerEventProcessor}} and then AM did not halt and did
> not log the following.Finally AM got hang since it's not thrown to main
> thread.
> {code}LOG.info("Halt with status " + status + " Message: " + msg);{code}
> *org.apache.hadoop.util.ExitUtil.halt(int, String)*
> {code}
> public static void halt(int status, String msg) throws HaltException {
> LOG.info("Halt with status " + status + " Message: " + msg);
> if (systemHaltDisabled) {
> HaltException ee = new HaltException(status, msg);
> LOG.fatal("Halt called", ee);
> if (null == firstHaltException) {
> firstHaltException = ee;
> }
> throw ee;
> }
> Runtime.getRuntime().halt(status);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6867) ApplicationMaster hung on OOM Error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bilwa S T updated MAPREDUCE-6867:
-
Attachment: MAPREDUCE-6867.patch
> ApplicationMaster hung on OOM Error
> ---
>
> Key: MAPREDUCE-6867
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6867
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Bilwa S T
>Assignee: Bilwa S T
> Attachments: MAPREDUCE-6867.patch
>
>
> Whenever OOM Error is thown, {{YarnUncaughtExceptionHandler}} will call
> {{ExitUtil.halt(-1)}}.But while halting, OOM might occur which is not
> handled.
> We came across a scenario where in when we submit mapreduce application ,OOM
> error occured in {{committerEventProcessor}} and then AM did not halt and did
> not log the following.Finally AM got hang since it's not thrown to main
> thread.
> {code}LOG.info("Halt with status " + status + " Message: " + msg);{code}
> *org.apache.hadoop.util.ExitUtil.halt(int, String)*
> {code}
> public static void halt(int status, String msg) throws HaltException {
> LOG.info("Halt with status " + status + " Message: " + msg);
> if (systemHaltDisabled) {
> HaltException ee = new HaltException(status, msg);
> LOG.fatal("Halt called", ee);
> if (null == firstHaltException) {
> firstHaltException = ee;
> }
> throw ee;
> }
> Runtime.getRuntime().halt(status);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6824) TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
[
https://issues.apache.org/jira/browse/MAPREDUCE-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-6824:
-
Resolution: Fixed
Fix Version/s: 2.9.0
Status: Resolved (was: Patch Available)
Committed this to branch-2. Thanks [~ctrezzo] for providing the patch!
> TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
> ---
>
> Key: MAPREDUCE-6824
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6824
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Trivial
> Labels: newbie
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: MAPREDUCE-6824-branch-2.001.patch,
> MAPREDUCE-6824-trunk.001.patch, MAPREDUCE-6824-trunk.002.patch,
> MAPREDUCE-6824-trunk.003.patch
>
>
> bq.
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752:
> private static ContainerLaunchContext createCommonContainerLaunchContext(:3:
> Method length is 172 lines (max allowed is 150).
> {{TaskAttemptImpl#createCommonContainerLaunchContext}} is longer than 150
> lines and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6824) TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
[
https://issues.apache.org/jira/browse/MAPREDUCE-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956460#comment-15956460
]
Akira Ajisaka commented on MAPREDUCE-6824:
--
+1, checking this in.
> TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
> ---
>
> Key: MAPREDUCE-6824
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6824
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Trivial
> Labels: newbie
> Fix For: 3.0.0-alpha3
>
> Attachments: MAPREDUCE-6824-branch-2.001.patch,
> MAPREDUCE-6824-trunk.001.patch, MAPREDUCE-6824-trunk.002.patch,
> MAPREDUCE-6824-trunk.003.patch
>
>
> bq.
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752:
> private static ContainerLaunchContext createCommonContainerLaunchContext(:3:
> Method length is 172 lines (max allowed is 150).
> {{TaskAttemptImpl#createCommonContainerLaunchContext}} is longer than 150
> lines and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
