[
https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771233#comment-15771233
]
Xiaobing Zhou commented on HDFS-10917:
--------------------------------------
Thanks [~xyao]. I posted v5 patch to address your comments #1, 2 and 3.
Only tracking the latency of sending packet to the last node in pipeline is a
conscious design choice.
In the case of pipeline [dn0, dn1, dn2], 5ms latency from dn0 to dn1, 100ms
from dn1 to dn2, NameNode claims dn2 is slow since it sees 100ms latency to
dn2. Note that NameNode is not ware of pipeline structure in this context and
only sees latency between two DataNodes.
In another case of the same pipeline, 100ms latency from dn0 to dn1, 5ms from
dn1 to dn2, NameNode will miss detecting dn1 being slow since it's not the last
node. However the assumption is that in a busy enough cluster there are many
other pipelines where dn1 is the last node, e.g. [dn3, dn4, dn1]. Also our
tracking interval is relatively long enough (at least an hour) to improve the
chances of the bad DataNodes being the last nodes in multiple pipelines.
> Collect peer performance statistics on DataNode.
> ------------------------------------------------
>
> Key: HDFS-10917
> URL: https://issues.apache.org/jira/browse/HDFS-10917
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 2.8.0
> Reporter: Xiaobing Zhou
> Assignee: Xiaobing Zhou
> Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch,
> HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch,
> HDFS-10917.005.patch
>
>
> DataNodes already detect if replication pipeline operations are slow and log
> warnings. For the purpose of analysis, performance metrics are desirable.
> This proposes adding them on DataNodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
