[
https://issues.apache.org/jira/browse/HDFS-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764311#comment-13764311
]
Eric Sirianni commented on HDFS-5183:
-------------------------------------
Copying over my feedback from HDFS-5157:
bq. I would favor a model like #2 but allow for the DataNode to override the
placement decision (where the override was likely only done in exceptional
circumstances).
This seems consistent with the overarching design of HDFS to allow for loose
synchronization in replica maps between the NameNode and DataNode
(re-synchronized periodically by block reports).
In particular, if the DataNode cannot honor the specific StorageID chosen by
the NameNode (but can still honor the write), I believe this should *not* be an
error condition. Do you all agree?
> Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to have a
> global view in choosing DN storage for replica.
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-5183
> URL: https://issues.apache.org/jira/browse/HDFS-5183
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode, performance
> Affects Versions: Heterogeneous Storage (HDFS-2832)
> Reporter: Junping Du
>
> Per discussion in HDFS-5157, There are two different ways to handle
> BlockPlacementPolicy and ReplicaChoosingPolicy in case of multiple storage
> types:
> 1. Client specifies the required storage type when calling addBlock(..) to
> NN. BlockPlacementPolicy in NN chooses a set of datanodes accounting for the
> storage type. Then, client passes the required storage type to the datanode
> set and each datanode chooses a particular storage using a
> VolumeChoosingPolicy.
> 2. Same as before, client specifies the required storage type when calling
> addBlock(..) to NN. Now, BlockPlacementPolicy in NN chooses a set of storages
> (instead of datanodes). Then, client writes to the corresponding storages.
> VolumeChoosingPolicy is no longer needed and it should be removed.
> We think #2 is more powerful as it will bring global view to volume choosing
> or bring storage status into consideration in replica choosing, so we propose
> to combine two polices together.
> One concern here is it may increase the load of NameNode as previously volume
> choosing is decided by DN. We may verify it later (that's why I put
> performance in component).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
