[
https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16188801#comment-16188801
]
Chris Trezzo edited comment on YARN-1492 at 10/2/17 8:50 PM:
-------------------------------------------------------------
[~asuresh] [~subru] I have set the target version for this jira back to 2.9.0.
The only jira that is left for this first phase is the documentation patch
(YARN-2960) and the startup script patch (YARN-4858). Both should be able to
make 2.9.0. The rest of the feature is already in branch-2. I have split out
some of the major features that still need to be finished in the shared cache
into a phase 2 jira (YARN-7282). That being said, the core parts of this
feature are committed and ready to be used in deployments that do not need
phase 2 features.
was (Author: ctrezzo):
[~asuresh] [~subru] I have set the target version for this jira back to 2.9.0.
The only jira that is left for this first phase is the documentation patch and
YARN-4858. Both should be able to make 2.9.0. The rest of the feature is
already in branch-2. I have split out some of the major features that still
need to be finished in the shared cache into a phase 2 jira (YARN-7282). That
being said, the core parts of this feature are committed and ready to be used
in deployments that do not need phase 2 features.
> truly shared cache for jars (jobjar/libjar)
> -------------------------------------------
>
> Key: YARN-1492
> URL: https://issues.apache.org/jira/browse/YARN-1492
> Project: Hadoop YARN
> Issue Type: New Feature
> Affects Versions: 2.0.4-alpha
> Reporter: Sangjin Lee
> Assignee: Chris Trezzo
> Attachments: shared_cache_design.pdf, shared_cache_design_v2.pdf,
> shared_cache_design_v3.pdf, shared_cache_design_v4.pdf,
> shared_cache_design_v5.pdf, shared_cache_design_v6.pdf,
> YARN-1492-all-trunk-v1.patch, YARN-1492-all-trunk-v2.patch,
> YARN-1492-all-trunk-v3.patch, YARN-1492-all-trunk-v4.patch,
> YARN-1492-all-trunk-v5.patch
>
>
> Currently there is the distributed cache that enables you to cache jars and
> files so that attempts from the same job can reuse them. However, sharing is
> limited with the distributed cache because it is normally on a per-job basis.
> On a large cluster, sometimes copying of jobjars and libjars becomes so
> prevalent that it consumes a large portion of the network bandwidth, not to
> speak of defeating the purpose of "bringing compute to where data is". This
> is wasteful because in most cases code doesn't change much across many jobs.
> I'd like to propose and discuss feasibility of introducing a truly shared
> cache so that multiple jobs from multiple users can share and cache jars.
> This JIRA is to open the discussion.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
