[
https://issues.apache.org/jira/browse/HADOOP-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13745321#comment-13745321
]
Jason Lowe commented on HADOOP-9639:
------------------------------------
Thanks for posting the design doc, Sangjin! A few comments based on a cursory
review:
* How does the client determine which job submission files are eligible for
participating in the public cache?
* The algorithm refers to a file being uploaded and "fully written." If files
are only renamed to the advertised location after it has finished uploading to
a temporary location first, how does a final name end up not being "fully
written"?
* Speaking of temporary files, there probably should be an algorithm clients
should use to try to derive a unique temporary name to avoid collisions,
similar to the mechanism to derive a unique reader lock. Bonus points if the
cleaner service recognizes and acts upon orphaned temporary files from clients
that crashed/disconnected during upload.
* When a client uploads a new file, loses the race with another client, and
determines for some reason that the version that won is bad, do we really want
that client removing the file? That seems like it will break the job that won
the race.
* Many workflows use "fire-and-forget" clients that do not have the job
submission client waiting for job completion. In that case the read lock will
be leaked by the client and the cleaner must remove any read lock files that
are considered stale even if it finds there is an active reference to the cache
entry. Failure to do so for a consistently active entry in the cache leads to
unbounded namespace leaks.
> truly shared cache for jars (jobjar/libjar)
> -------------------------------------------
>
> Key: HADOOP-9639
> URL: https://issues.apache.org/jira/browse/HADOOP-9639
> Project: Hadoop Common
> Issue Type: New Feature
> Components: filecache
> Affects Versions: 2.0.4-alpha
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Attachments: shared_cache_design.pdf
>
>
> Currently there is the distributed cache that enables you to cache jars and
> files so that attempts from the same job can reuse them. However, sharing is
> limited with the distributed cache because it is normally on a per-job basis.
> On a large cluster, sometimes copying of jobjars and libjars becomes so
> prevalent that it consumes a large portion of the network bandwidth, not to
> speak of defeating the purpose of "bringing compute to where data is". This
> is wasteful because in most cases code doesn't change much across many jobs.
> I'd like to propose and discuss feasibility of introducing a truly shared
> cache so that multiple jobs from multiple users can share and cache jars.
> This JIRA is to open the discussion.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
