[
https://issues.apache.org/jira/browse/HDFS-8182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504105#comment-14504105
]
Andrew Wang commented on HDFS-8182:
-----------------------------------
Hi Gera, just for my own edification, given the "spread" block placement
policy, is it not sufficient to set the distributed cache repl factor to the #
of racks in the cluster? Is the space overhead too much?
It's hard for me to evaluate as I don't have much familiarity with dist cache,
e.g. typical # bytes in dist cache, # racks, dist cache repl factor. If the
overhead ends up being tens of GBs per additional rack though, that seems
acceptable considering a rack holds a PB+ these days.
> Implement topology-aware CDN-style caching
> ------------------------------------------
>
> Key: HDFS-8182
> URL: https://issues.apache.org/jira/browse/HDFS-8182
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
> Affects Versions: 2.6.0
> Reporter: Gera Shegalov
>
> To scale reads of hot blocks in large clusters, it would be beneficial if we
> could read a block across the ToR switches only once. Example scenarios are
> localization of binaries, MR distributed cache files for map-side joins and
> similar. There are multiple layers where this could be implemented (YARN
> service or individual apps such as MR) but I believe it is best done in HDFS
> or even common FileSystem to support as many use cases as possible.
> The life cycle could look like this e.g. for the YARN localization scenario:
> 1. inputStream = fs.open(path, ..., CACHE_IN_RACK)
> 2. instead of reading from a remote DN directly, NN tells the client to read
> via the local DN1 and the DN1 creates a replica of each block.
> When the next localizer on DN2 in the same rack starts it will learn from NN
> about the replica in DN1 and the client will read from DN1 using the
> conventional path.
> When the application ends the AM or NM's can instruct the NN in a fadvise
> DONTNEED style, it can start telling DN's to discard extraneous replica.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
