[
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989035#comment-15989035
]
Erik Krogen commented on MAPREDUCE-5951:
----------------------------------------
Ah, excellent point, [~jlowe]... I actually would love to hear the reasoning
behind the current strategy of <client uploads resource to HDFS -> AM downloads
resource -> AM uploads resource to SCM> rather than the seemingly more
obvious/simpler <client uploads resource to SCM>. Is this so that the uploading
to SCM can be done by the NM, which is a privileged user, to have more secure
control over it?
[~ctrezzo], first off thanks for getting back so quickly! And for the pointer
to YARN-5727; that's an interesting issue. The public visibility solution is
certainly simpler from the YARN side and seems pretty reasonable from a point
of expectation of burden on an application ("you want a publicly shared
resource? put it somewhere public"). It doesn't add _too_ much complexity on
the MR side, though having a separate staging directory just for public
resources is a bit cumbersome. It also means that other application developers
will have to build the same type of logic - in general I would lean towards
more logic pushed into the YARN level so that it is easy for application devs
to support. I don't have good insight into how difficult your initially
proposed solution in YARN-5727 would be to implement, though.
> Add support for the YARN Shared Cache
> -------------------------------------
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Chris Trezzo
> Assignee: Chris Trezzo
> Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf,
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch,
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch,
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch,
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch,
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch,
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch,
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch,
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch,
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch,
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify
> which set of resources they would like to cache (i.e. jobjar, libjars,
> archives, files).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
