[
https://issues.apache.org/jira/browse/HDFS-5149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790950#comment-13790950
]
Chris Nauroth commented on HDFS-5149:
-------------------------------------
Hi [~andrew.wang]. Do you already have something specific in mind for the
scope of this issue? Here are a few of my initial thoughts based on current
state of the code in the HDFS-4949 branch:
Periodic cache reports sent from the datanode already include block pool ID and
are composed of the cached blocks in that specific pool.
({{BPServiceActor#cacheReport}}) I don't see any additional work required on
the protocol here.
{{FsDatasetCache}} operations are already implemented with consideration of
block pool ID. It's possible that there is room for optimization so that e.g.
{{FsDatasetCache#getCachedBlocks}} doesn't need to do a full scan to filter by
block pool ID. (I believe this is just optimization work in the implementation
though, and not a question of correctness.)
The namenode ignores the block pool ID that was sent to it.
({{CacheReplicationManager#processCacheReport}}) The existing {{BlockManager}}
similarly ignores block pool ID. Again, perhaps there is optimization
opportunity here if the data structures were indexed by block pool ID, but I
don't see an issue of correctness in a federated deployment.
Metrics are not available on a per-block-pool basis (i.e. no
{{FsDatasetImpl#getBlockPoolCacheUsed}}). We could potentially add this.
We probably need testing of adding a new namenode to an existing cluster to
ensure that caching works correctly in the newly added namenode.
What are your thoughts?
> Support for federated cache pools
> ---------------------------------
>
> Key: HDFS-5149
> URL: https://issues.apache.org/jira/browse/HDFS-5149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Affects Versions: HDFS-4949
> Reporter: Andrew Wang
> Assignee: Andrew Wang
>
> The federation concept of a "blockpool" has not pervaded the caching
> implementation thus far, e.g. datanodes do not track and report usage by
> block pool, some namenode operations are not block pool aware. This requires
> re-examination of protobufs / protocol / handling throughout.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
