[
https://issues.apache.org/jira/browse/HADOOP-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13745431#comment-13745431
]
Sangjin Lee commented on HADOOP-9639:
-------------------------------------
{quote}
How does the client determine which job submission files are eligible for
participating in the public cache?
{quote}
Right now, it is a binary choice. If the binary key is set, the job jar and
libjars (if any) will be all sharable/shared. However, with the APIs you should
be able to have a finer-grained control. Is that acceptable? Could you give me
a scenario under which the client may want a finer-grained control? Is that
what you were getting at?
{quote}
The algorithm refers to a file being uploaded and "fully written." If files are
only renamed to the advertised location after it has finished uploading to a
temporary location first, how does a final name end up not being "fully
written"?
{quote}
Maybe the term "fully written" is bit misleading. It is really a defensive
check to guard against an *erroneous* file. You're right if the file is
uploaded via client with this functionality, it shouldn't be possible. This is
more about guarding against bad files.
{quote}
Speaking of temporary files, there probably should be an algorithm clients
should use to try to derive a unique temporary name to avoid collisions,
similar to the mechanism to derive a unique reader lock. Bonus points if the
cleaner service recognizes and acts upon orphaned temporary files from clients
that crashed/disconnected during upload.
{quote}
Agreed on having a common (or similar) algorithm for deriving unique names.
I've thought about cleaning up orphaned temporary files. The difficulty is to
determine authoritatively whether a certain temporary file is truly unused. One
could check whether the file is closed. But note that a closed temp file may be
in use (as a client may be using a temp file and it's being localized, etc.).
Otherwise, heuristics may become hairy; e.g. "old enough" (how old is old
enough?), etc. One mitigating factor is that the occurrence of orphaned
temporary files would be pretty uncommon (unlike the orphaned reader lock
files).
{quote}
When a client uploads a new file, loses the race with another client, and
determines for some reason that the version that won is bad, do we really want
that client removing the file? That seems like it will break the job that won
the race.
{quote}
I believe the file should be necessarily a bad one in this situation. For this
to happen, the size of the uploaded jar should not match what I have. However,
given the right checksum, the size cannot be different. I had this step here to
clean up the bad jar so that other clients can "repair" the bad jar.
{quote}
Many workflows use "fire-and-forget" clients that do not have the job
submission client waiting for job completion. In that case the read lock will
be leaked by the client and the cleaner must remove any read lock files that
are considered stale even if it finds there is an active reference to the cache
entry. Failure to do so for a consistently active entry in the cache leads to
unbounded namespace leaks.
{quote}
Yes, that's a good point. We thought about this scenario (see the tasks
section), and I think it should be safe to have the cleaner service remove all
reader locks except for the latest one. I'll consider doing that. We just need
to be careful in selecting the directories to clean in this manner as removing
these files would also update the directory modification time and "refresh" the
directory artificially.
> truly shared cache for jars (jobjar/libjar)
> -------------------------------------------
>
> Key: HADOOP-9639
> URL: https://issues.apache.org/jira/browse/HADOOP-9639
> Project: Hadoop Common
> Issue Type: New Feature
> Components: filecache
> Affects Versions: 2.0.4-alpha
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Attachments: shared_cache_design.pdf
>
>
> Currently there is the distributed cache that enables you to cache jars and
> files so that attempts from the same job can reuse them. However, sharing is
> limited with the distributed cache because it is normally on a per-job basis.
> On a large cluster, sometimes copying of jobjars and libjars becomes so
> prevalent that it consumes a large portion of the network bandwidth, not to
> speak of defeating the purpose of "bringing compute to where data is". This
> is wasteful because in most cases code doesn't change much across many jobs.
> I'd like to propose and discuss feasibility of introducing a truly shared
> cache so that multiple jobs from multiple users can share and cache jars.
> This JIRA is to open the discussion.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
