[jira] [Commented] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort.md
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719869#comment-16719869
]
Hudson commented on MAPREDUCE-7170:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15602 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15602/])
MAPREDUCE-7170. Doc typo in PluggableShuffleAndPluggableSort.md. (aajisaka: rev
e0d75088f24283985a4e4032fa547dad2ebb54db)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md
> Doc typo in PluggableShuffleAndPluggableSort.md
> ---
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: documentation
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Fix For: 3.1.2, 3.3.0, 3.2.1
>
> Attachments: MAPREDUCE-7170.001.patch, MAPREDUCE-7170.002.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort.md
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7170:
-
Resolution: Fixed
Fix Version/s: 3.2.1
3.3.0
3.1.2
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-3.2, and branch-3.1. Thanks [~uranus] for the
contribution!
> Doc typo in PluggableShuffleAndPluggableSort.md
> ---
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: documentation
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Fix For: 3.1.2, 3.3.0, 3.2.1
>
> Attachments: MAPREDUCE-7170.001.patch, MAPREDUCE-7170.002.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort.md
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7170:
-
Summary: Doc typo in PluggableShuffleAndPluggableSort.md (was: Doc
typo in PluggableShuffleAndPluggableSort)
Component/s: documentation
> Doc typo in PluggableShuffleAndPluggableSort.md
> ---
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: documentation
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Attachments: MAPREDUCE-7170.001.patch, MAPREDUCE-7170.002.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7170:
-
Hadoop Flags: Reviewed
Issue Type: Bug (was: Improvement)
+1
> Doc typo in PluggableShuffleAndPluggableSort
>
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Attachments: MAPREDUCE-7170.001.patch, MAPREDUCE-7170.002.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719807#comment-16719807
]
Hadoop QA commented on MAPREDUCE-7170:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
31m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | MAPREDUCE-7170 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951606/MAPREDUCE-7170.002.patch
|
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux d03b0d0b989d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 72e7c6a |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 443 (vs. ulimit of 1) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7564/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Doc typo in PluggableShuffleAndPluggableSort
>
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Attachments: MAPREDUCE-7170.001.patch, MAPREDUCE-7170.002.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719778#comment-16719778
]
Akira Ajisaka commented on MAPREDUCE-7170:
--
LGTM, +1 pending Jenkins. Thanks!
> Doc typo in PluggableShuffleAndPluggableSort
>
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Attachments: MAPREDUCE-7170.001.patch, MAPREDUCE-7170.002.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhaohui Xin updated MAPREDUCE-7170:
---
Attachment: MAPREDUCE-7170.002.patch
> Doc typo in PluggableShuffleAndPluggableSort
>
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Attachments: MAPREDUCE-7170.001.patch, MAPREDUCE-7170.002.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719766#comment-16719766
]
Zhaohui Xin commented on MAPREDUCE-7170:
L109 is also fixed. Added new patch. :D
> Doc typo in PluggableShuffleAndPluggableSort
>
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Attachments: MAPREDUCE-7170.001.patch, MAPREDUCE-7170.002.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7170:
-
Priority: Minor (was: Major)
Target Version/s: 3.1.2, 3.3.0, 3.2.1
> Doc typo in PluggableShuffleAndPluggableSort
>
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Attachments: MAPREDUCE-7170.001.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7170) Doc typo in PluggableShuffleAndPluggableSort
[
https://issues.apache.org/jira/browse/MAPREDUCE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719702#comment-16719702
]
Akira Ajisaka commented on MAPREDUCE-7170:
--
Would you fix L109 as well?
> Doc typo in PluggableShuffleAndPluggableSort
>
>
> Key: MAPREDUCE-7170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7170
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Major
> Attachments: MAPREDUCE-7170.001.patch
>
>
>
> {code:java}
>
> yarn.nodemanager.aux-services.AuxServiceFromHDFS.class
> org.apache.auxtest.AuxServiceFromHDFS2
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7172) Wildcard functionality of -libjar is broken when jars are located in same remote FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719260#comment-16719260
]
Wangda Tan commented on MAPREDUCE-7172:
---
Thanks [~templedf] for trying this.
Per my understanding, the code flow is:
Inside org.apache.hadoop.mapreduce.JobResourceUploader#uploadLibJars
1) Mkdir for the job submission libjar dir (on remote FS path):
{code}
Path libjarsDir = JobSubmissionFiles.getJobDistCacheLibjars(submitJobDir);
{code}
2) Copy remote files:
{code}
if (newURI == null) {
Path newPath =
copyRemoteFiles(libjarsDir, tmp, conf, submitReplication);
{code}
Important: shared cache libjar should be disabled:
{code}
if (scConfig.isSharedCacheLibjarsEnabled()) {
{code}
3) Inside copyRemoteFiles:
{code}
if (FileUtil.compareFs(remoteFs, jtFs)) {
return originalPath;
}
{code}
If remoteFS == jtFS, nothing will be copied. And returns originalPath. (So the
returned newPath is same as originalPath).
4) Call addFileToClasspath, but last parameter will be false. (So file will be
added to classpath, but won't be downloaded by NM).
{code}
DistributedCache.addFileToClassPath(new Path(newURI.getPath()), conf,
jtFs, false);
{code}
5) When wildcard is enabled, and no fragment (please make sure no fragment).
{code}
if (useWildcard && !foundFragment) {
// Add the whole directory to the cache using a wild card
Path libJarsDirWildcard =
jtFs.makeQualified(new Path(libjarsDir, DistributedCache.WILDCARD));
DistributedCache.addCacheFile(libJarsDirWildcard.toUri(), conf);
}
{code}
The whole remote libjar directory will be localized. But because of 3), files
won't be copied to remote libjar.
I will try to spend some time today to reproduce this issue on a local cluster,
if you have time, could u check my comment as well? :).
> Wildcard functionality of -libjar is broken when jars are located in same
> remote FS
> ---
>
> Key: MAPREDUCE-7172
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7172
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Wangda Tan
>Priority: Critical
>
> We recently found that when -libjar specified jars on the same remote FS,
> jars will not be properly added to classpath.
> The reason is MAPREDUCE-6719 added the wildcard functionality, but the follow
> logic assumes files are all placed under job's submission directory. (Inside
> JobResourceUploader)
> {code:java}
> if (useWildcard && !foundFragment) {
> // Add the whole directory to the cache using a wild card
> Path libJarsDirWildcard =
> jtFs.makeQualified(new Path(libjarsDir, DistributedCache.WILDCARD));
> DistributedCache.addCacheFile(libJarsDirWildcard.toUri(), conf);
> }{code}
> However, in the same method, specified resources will be only uploaded when
> two FSes are different, see copyRemoteFiles:
> {code:java}
> if (FileUtil.compareFs(remoteFs, jtFs)) {
> return originalPath;
> } {code}
> Workaround of this issue is pass:
> mapreduce.client.libjars.wildcard = false.
> When the MR job got launched.
> Example commandline to reproduce this issue is:
> {code:java}
> hadoop jar abc.jar org.ABC -libjars
> "wasb://host/path1/jar1,wasb://host/path2/jar2..."{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7172) Wildcard functionality of -libjar is broken when jars are located in same remote FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719005#comment-16719005
]
Daniel Templeton commented on MAPREDUCE-7172:
-
Playing around with a cluster a bit, I don't see an issue. I setup a
single-node cluster using trunk. I submitted a job with two libjars as
suggested by [~leftnoteasy] above. I can see that wild card paths are enabled
in the job.xml. I can see that both libjars are uploaded to the libjars
directory in the job's staging directory and that both libjars are downloaded
into the job's working directory on the node.
Looking at the source code, I also fail to see what the issue is.
[~leftnoteasy], what am I missing?
> Wildcard functionality of -libjar is broken when jars are located in same
> remote FS
> ---
>
> Key: MAPREDUCE-7172
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7172
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Wangda Tan
>Priority: Critical
>
> We recently found that when -libjar specified jars on the same remote FS,
> jars will not be properly added to classpath.
> The reason is MAPREDUCE-6719 added the wildcard functionality, but the follow
> logic assumes files are all placed under job's submission directory. (Inside
> JobResourceUploader)
> {code:java}
> if (useWildcard && !foundFragment) {
> // Add the whole directory to the cache using a wild card
> Path libJarsDirWildcard =
> jtFs.makeQualified(new Path(libjarsDir, DistributedCache.WILDCARD));
> DistributedCache.addCacheFile(libJarsDirWildcard.toUri(), conf);
> }{code}
> However, in the same method, specified resources will be only uploaded when
> two FSes are different, see copyRemoteFiles:
> {code:java}
> if (FileUtil.compareFs(remoteFs, jtFs)) {
> return originalPath;
> } {code}
> Workaround of this issue is pass:
> mapreduce.client.libjars.wildcard = false.
> When the MR job got launched.
> Example commandline to reproduce this issue is:
> {code:java}
> hadoop jar abc.jar org.ABC -libjars
> "wasb://host/path1/jar1,wasb://host/path2/jar2..."{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
