Chris Trezzo updated YARN-1492:

    Attachment: YARN-1492-all-trunk-v1.patch

Attached is an all inclusive trunk patch for the shared cache code. I will 
break it up into smaller more reviewable patches and attach them to subtasks. 
There are three major components to the yarn shared cache (these are in-line 
with the posted design document):
1. The shared cache manager - this is a new service responsible for maintaining 
the shared cache metadata (i.e. what is in the cache and which applications are 
currently using each entry). The manager currently leverages an in-memory 
store. In the case of a manager restart, the cache entries are recreated from 
2. Client side changes - These include changes at the yarn and MapReduce 
layers. There is a new yarn shared cache api and changes to the MapReduce 
JobSubmitter to leverage the new api.
3. Upload Service - There is a new service in the NodeManager that is 
responsible for uploading resources to the shared cache.

> truly shared cache for jars (jobjar/libjar)
> -------------------------------------------
>                 Key: YARN-1492
>                 URL: https://issues.apache.org/jira/browse/YARN-1492
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.0.4-alpha
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: YARN-1492-all-trunk-v1.patch, shared_cache_design.pdf, 
> shared_cache_design_v2.pdf, shared_cache_design_v3.pdf, 
> shared_cache_design_v4.pdf, shared_cache_design_v5.pdf
> Currently there is the distributed cache that enables you to cache jars and 
> files so that attempts from the same job can reuse them. However, sharing is 
> limited with the distributed cache because it is normally on a per-job basis. 
> On a large cluster, sometimes copying of jobjars and libjars becomes so 
> prevalent that it consumes a large portion of the network bandwidth, not to 
> speak of defeating the purpose of "bringing compute to where data is". This 
> is wasteful because in most cases code doesn't change much across many jobs.
> I'd like to propose and discuss feasibility of introducing a truly shared 
> cache so that multiple jobs from multiple users can share and cache jars. 
> This JIRA is to open the discussion.

This message was sent by Atlassian JIRA

Reply via email to