[
https://issues.apache.org/jira/browse/MAPREDUCE-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719005#comment-16719005
]
Daniel Templeton commented on MAPREDUCE-7172:
---------------------------------------------
Playing around with a cluster a bit, I don't see an issue. I setup a
single-node cluster using trunk. I submitted a job with two libjars as
suggested by [~leftnoteasy] above. I can see that wild card paths are enabled
in the job.xml. I can see that both libjars are uploaded to the libjars
directory in the job's staging directory and that both libjars are downloaded
into the job's working directory on the node.
Looking at the source code, I also fail to see what the issue is.
[~leftnoteasy], what am I missing?
> Wildcard functionality of -libjar is broken when jars are located in same
> remote FS
> -----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7172
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7172
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Wangda Tan
> Priority: Critical
>
> We recently found that when -libjar specified jars on the same remote FS,
> jars will not be properly added to classpath.
> The reason is MAPREDUCE-6719 added the wildcard functionality, but the follow
> logic assumes files are all placed under job's submission directory. (Inside
> JobResourceUploader)
> {code:java}
> if (useWildcard && !foundFragment) {
> // Add the whole directory to the cache using a wild card
> Path libJarsDirWildcard =
> jtFs.makeQualified(new Path(libjarsDir, DistributedCache.WILDCARD));
> DistributedCache.addCacheFile(libJarsDirWildcard.toUri(), conf);
> }{code}
> However, in the same method, specified resources will be only uploaded when
> two FSes are different, see copyRemoteFiles:
> {code:java}
> if (FileUtil.compareFs(remoteFs, jtFs)) {
> return originalPath;
> } {code}
> Workaround of this issue is pass:
> mapreduce.client.libjars.wildcard = false.
> When the MR job got launched.
> Example commandline to reproduce this issue is:
> {code:java}
> hadoop jar abc.jar org.ABC -libjars
> "wasb://host/path1/jar1,wasb://host/path2/jar2..."{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
