[
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated MAPREDUCE-5951:
------------------------------------
Attachment: MAPREDUCE-5951-trunk-v10.patch
V10 Attached.
1. This patch takes out support for symlinking of two cached resources with the
same name. [~kasha] and I chatted offline and this seems like something complex
enough that it should be handled for all shared cache clients at the YARN
layer. Note: Because of this change, this patch will currently not handle
resources correctly in certain shared cache scenarios. Please see YARN-3637 for
more context.
bq. JobImpl - is the cleanup of upload-policies intended to be in
init-transition? Is that because we don't need the policies once we are done
uploading the resources?
2. The cleanup of upload-policies happens in the Application Master
init-transition to prevent all node managers running tasks for this job from
attempting to upload resources to the shared cache. Since all containers in the
MapReduce application localize the same resources, we decided that the
Application Master is the only container that needs to upload resources. Maybe
this needs more fool proofing to prevent redundant resource upload attempts?
3. The order of isScmAvailable() check in isSharedCacheFilesEnabled and co. was
changed.
4. Renamed getFiles to mergeLocalAndCacheResources and made it static.
> Add support for the YARN Shared Cache
> -------------------------------------
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Chris Trezzo
> Assignee: Chris Trezzo
> Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-trunk-v1.patch,
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v2.patch,
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch,
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch,
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch,
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify
> which set of resources they would like to cache (i.e. jobjar, libjars,
> archives, files).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
