[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11530:
------------------------------
Attachment: HDFS-11530.003.patch
Hi [~linyiqun] I think I've identified one issue, which is that the code does
not handle heartbeat. Specifically, it appears to me that it works in the way
that a node is added first, but with an empty storage info at this point. Later
when the first heartbeat comes, {{DatanodeDescriptor}} gets updated and the
storage info gets filled. Namely, {{DFSTopologyNodeImpl}} needs to be updated
accordingly on receiving heartbeats, this is the missing piece causing at least
part of the problem. Also, adding the heart-triggered-updates might just handle
datanode failover situation, but this needs to be verified.
Post v003 patch here to add updates of storage types on heartbeat. The high
level idea is that when a heartbeat indicates a datanode has storage type
add/remove, inform the parent recursively to have all the counters updated.
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
> ------------------------------------------------------------------------------------
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Affects Versions: 3.0.0-alpha2
> Reporter: Yiqun Lin
> Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
