[
https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725704#comment-13725704
]
David S. Wang commented on HDFS-4949:
-------------------------------------
HDFS-4949 meeting, July 29, 2013 2 PM @ Hortonworks office
====
Attendees
====
Aaron T. Myers
Andrew Wang
Arpit Gupta
Bikas Saha
Brandon Li
Colin McCabe
Dave Wang
Jing Zhao
Suresh Srinivas
Sanjay Radia
Todd Lipcon
Vinod Kumar Vavilapalli
Minutes
====
* General agreement to hold HDFS-2832 meeting some other day
* Andrew: Posted HDFS-4949 design doc upstream; Sanjay has read this, agrees
with the goals
Data path (zero-copy reads)
----
* Sanjay: quota mgmt - counted up front, not after cache is populated
* Colin: talking about ZCR (mmap) - used to implement caching at the DN level
** Considered copying everything into /dev/shm (e.g. Tachyon). But cannot
cache parts of a file, so limits our flexibility. Also, the associated fd
gives clients a way to control memory mgmt (will not release until that
descriptor is closed), which is not good because of buggy clients etc.
** Sanjay: you want an abstraction for a durable file. Colin: yes.
** Colin: ZCR currently doesn't have checksums, but will. Todd: assumption is
that DN will do the cksum when doing the mlock and communicate that to the
client so the client knows that it's safe to read.
** Todd: mincore() can tell you what's already in cache, but it's too granular,
very expensive to call, and can be out-of-date immediately.
** Assuming this is for local clients only obviously.
** ZCR uses ByteBuffers to avoid copies. Not entirely compatible with current
DFSClient since that uses byte arrays, so you cannot avoid copies.
* This may have a conflict with inline checksums. Clients would have to be
aware of how to skip over checksums, and this would have to be in the app, not
the client since we're talking mmap.
** HBase gets around this by disabling HDFS-level cksums, and doing it on their
level.
** Sanjay: QFS puts all of the cksums in the beginning of the file
** Todd: Liang Xie had an HBase study where he figured out that perf didn't
improve until he got to a TB of data, when the cksum files themselves dropped
out ofcache.
* ZCR API can be made public? Colin, Todd: Yes.
** Hard to compete with Spark if this isn't public.
** Suresh: Will the app know if you got ZCR? Can be added as counters. Colin:
already have similar concepts today for SCR on a per-stream basis.
** Todd: SCR is fully transparent (uses today's API), while ZCR requires new
client API.
* Sanjay: Current policy is manual. Later policy can have system automatically
cache hot files. Need the fallback buffer in case you are remote.
** Todd: high perf apps will always use the ZCR API. Sometimes it will fall
back to a normal read, so no worse than today.
** Colin: should we have a flag that basically says "always mmap"? Can add it
later, don't know how useful this could be.
* Colin: no native support required for ZCR beyond what is there today. There
are some libhdfs changes, but not completely required. Java has mmap today.
** We will need a native call for locking though.
Centralized cache mgmt
----
* Andrew gave whiteboard presentation
** DN has mlock hooks, ulimit conf of how much it can cache
** NN sends heartbeats to DN with cache/uncache commands on whole blocks
** DN will send cache state to NN similar to block reports
** clients call getFileBlockLocations() with storageType arg. This returns the
current state of the cache.
** clients can issue CachingRequests, with a path that points to a file or
directory. If directory, then what is cached is what's in that directory (but
not recurse to subdirs), in order to support Hive. Can also specify user, pool
for quote mgmt. Can also specify # cache copies (must be <= replicationFactor).
* Quotas
** Quotas are on pools, not users. Quotas enforced on the NN.
** Questions about what is cached as machines come and go? Use
getFileBlockLocations() to get cache request and current status of fulfillment.
Can be not fulfilled due to quota for instance.
*** Should cache requests from two pools be counted fully against both?
Half-half? Cluster capacity can be dynamic, so you always have potential quote
mgmt problems.
** Don't want to get this so complicated so that you basically are implementing
another scheduler just for cache quotas.
** Suresh: Resource failures - how does this affect the pools? Should we have
priorities for pools? Priorities for individual CacheRequests?
** Andrew: suggestion of min/max/share (similar to VMware ESX VM memory
configuration).
** Suresh: fine with doing something very basic, and then be more intelligent
later.
** Sanjay: need to have some idea of per-pool priority to enforce min, to
figure out what to evict from the cache first in mem-constrained scenarios.
Also what happens once we have resources again?
* Suresh: don't cache files being written to? ATM, others: yes.
* Cache entire files vs. shares of files in a request that can't be 100%
satisfied? Yes.
* Interaction with symlinks? Discuss on JIRA.
* Suresh: only cache something that is local to DN? Yes.
* Sanjay: should block reports should include storage types? ATM: Treat RAM as
a special case at least for now (e.g. have a bit that indicates that this
replica is in RAM).
* Sanjay: leases for the CacheRequests? Todd: CacheRequests could have their
status listed as a API call, not a lease. Otherwise clients have to keep
requesting renewals, which sucks for existing folks like Hive.
** Suresh: how about a expiry time in the CacheRequest?
* Sanjay: Question about how big the RPCs are from NN -> DN
** Todd: assuming 10 MB blocks, and 100 GB cache memory for a DN, then max of
10K blocks in an RPC. But will be smaller due to incremental reports.
** Suresh: perhaps push down idea of collection id to DN, so NN can just say
"evict this collection" instead of having to send evicts for all of the blocks.
Can be addressed later in a compatible fashion.
* Suresh: how can we help?
** Colin: review on the ZCR would be great.
** Andrew: HDFS-4949: will start churning on this ASAP, over the next month.
Could use some help with testing.
* Suresh: branch to put this in? 2.3? That's where all of the feature work is
going.
> Centralized cache management in HDFS
> ------------------------------------
>
> Key: HDFS-4949
> URL: https://issues.apache.org/jira/browse/HDFS-4949
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Affects Versions: 3.0.0, 2.3.0
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Attachments: caching-design-doc-2013-07-02.pdf
>
>
> HDFS currently has no support for managing or exposing in-memory caches at
> datanodes. This makes it harder for higher level application frameworks like
> Hive, Pig, and Impala to effectively use cluster memory, because they cannot
> explicitly cache important datasets or place their tasks for memory locality.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
