[
https://issues.apache.org/jira/browse/MAPREDUCE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342408#comment-15342408
]
Hudson commented on MAPREDUCE-6719:
-----------------------------------
SUCCESS: Integrated in Hadoop-trunk-Commit #9995 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9995/])
MAPREDUCE-6719. The list of -libjars archives should be replaced with a (sjlee:
rev 605b4b61364781fc99ed27035c793153a20d8f71)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestDistributedCache.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
> The list of -libjars archives should be replaced with a wildcard in the
> distributed cache to reduce the application footprint in the state store
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6719
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6719
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 2.8.0
> Reporter: Daniel Templeton
> Assignee: Daniel Templeton
> Priority: Critical
> Fix For: 2.9.0
>
> Attachments: MAPREDUCE-6719.001.patch, MAPREDUCE-6719.002.patch
>
>
> When using the -libjars option to add classes to the classpath, every library
> so added is explicitly listed in the ContainerLaunchContext's local resources
> even though they're all uploaded to the same directory in HDFS. When using
> tools like Crunch without an uber JAR or when trying to take advantage of the
> shared cache, the number of libraries can be quite large. We've seen many
> cases where we had to turn down the max number of applications to prevent ZK
> from running out of heap because of the size of the state store entries.
> This JIRA proposes to allow for wildcards both in the internal processing of
> the -libjars switch and in paths added through the Job and DistributedCache
> classes. Rather than listing all files independently, this JIRA proposes to
> replace the complete list of libdir files with the wildcarded libdir
> directory, e.g. "libdir/*". This behavior is the same as the current behavior
> when using -libjars, but avoids explicitly listing every file.
> This capability will also be exposed by the
> {{DistributedCache.addCacheFile()}} method.
> See YARN-4958 for the NM side of the implementation and additional discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
