[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920076#comment-13920076
]
Sanjay Radia commented on HDFS-5477:
------------------------------------
I read both documents; some key details are missing (perhaps in the patches,
but they need to be in the document or jira).
* How does the BM know the replication factor of a block? The document tends to
suggest that the BM syncs with the NN on its start. Is this a sort of a
"reverse Block report" from NN to BM where the NN tells the BM its list of
blocks and their replication factor?
** In particular, one would like the BM's state to exist independently of the
NN so that if one or more NNs are shut down for long periods of time (such as
unmounting a namespace) then blocks and their replicas are still managed. Would
this be possible under your design? If not what would it take to support it?
* How would one support file affinity? - for example there are several use
cases (esp for HBase) where one co-locates file blocks replicas of multiple
files together. How would you support such a feature?
* You briefly reference fine grained locking in the NN - does your design
require that the NN holds certain fine grained locks as threads in the NN makes
a remote call to the BM? Can you please detail these in context of specific
operations in the document and/or jira.
* How do you plan to handle file and block-under construction? There is
delicate code that finalizes sizes of block under construction especially in
face of NN and DN restarts?
* Please describe how you would support the Balancer in your new design.
* Your design does not change client APIs and hence you must be forwarding
block location requires via the NN to the BM. Can you describe how one could
direct clients to go directly to BM for block locations when appropriate (I
believe this can done in API compatible way and also, with protocol changes,
in a backward compatible way.)
> Block manager as a service
> --------------------------
>
> Key: HDFS-5477
> URL: https://issues.apache.org/jira/browse/HDFS-5477
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: Proposal.pdf, Proposal.pdf, Standalone BM.pdf,
> Standalone BM.pdf, patches.tar.gz
>
>
> The block manager needs to evolve towards having the ability to run as a
> standalone service to improve NN vertical and horizontal scalability. The
> goal is reducing the memory footprint of the NN proper to support larger
> namespaces, and improve overall performance by decoupling the block manager
> from the namespace and its lock. Ideally, a distinct BM will be transparent
> to clients and DNs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
