[
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated MAPREDUCE-5951:
------------------------------------
Attachment: MAPREDUCE-5951-trunk-v8.patch
[[email protected]]
Attached is v8 of the patch. This update did two things:
1. Added the DistributedCache changes back in. On second thought, they are
necessary for this patch. The key is the
DistributedCache#getPathStringWithoutFragment method. This method is added and
used during the construction of the classpath, so that the symlinking during
localization of resources works correctly in the case when there are two
different resources with the same name. For example, checksum1/job.jar,
checksum2/job.jar. If both of these resources are used as libjars in a single
job, there will be a naming conflict and one of the resource's container
symlink (on the node manager) will get overridden by the other. We use the
fragment portion of the URI to control the symlink used by yarn localization.
With this, we ensure to avoid naming conflicts at the container level (users
can name their jars however they want) and we leverage shared jars coming from
the cache.
2. Fixed TestJobResourceUploader unit tests to accommodate using
ConverterUtils#toApplicationId.
> Add support for the YARN Shared Cache
> -------------------------------------
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Chris Trezzo
> Assignee: Chris Trezzo
> Attachments: MAPREDUCE-5951-trunk-v1.patch,
> MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch,
> MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch,
> MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch,
> MAPREDUCE-5951-trunk-v8.patch
>
>
> Implement the necessary changes so that the MapReduce application can
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify
> which set of resources they would like to cache (i.e. jobjar, libjars,
> archives, files).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
