[ https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771233#comment-15771233 ]
Xiaobing Zhou commented on HDFS-10917: -------------------------------------- Thanks [~xyao]. I posted v5 patch to address your comments #1, 2 and 3. Only tracking the latency of sending packet to the last node in pipeline is a conscious design choice. In the case of pipeline [dn0, dn1, dn2], 5ms latency from dn0 to dn1, 100ms from dn1 to dn2, NameNode claims dn2 is slow since it sees 100ms latency to dn2. Note that NameNode is not ware of pipeline structure in this context and only sees latency between two DataNodes. In another case of the same pipeline, 100ms latency from dn0 to dn1, 5ms from dn1 to dn2, NameNode will miss detecting dn1 being slow since it's not the last node. However the assumption is that in a busy enough cluster there are many other pipelines where dn1 is the last node, e.g. [dn3, dn4, dn1]. Also our tracking interval is relatively long enough (at least an hour) to improve the chances of the bad DataNodes being the last nodes in multiple pipelines. > Collect peer performance statistics on DataNode. > ------------------------------------------------ > > Key: HDFS-10917 > URL: https://issues.apache.org/jira/browse/HDFS-10917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Affects Versions: 2.8.0 > Reporter: Xiaobing Zhou > Assignee: Xiaobing Zhou > Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch, > HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch, > HDFS-10917.005.patch > > > DataNodes already detect if replication pipeline operations are slow and log > warnings. For the purpose of analysis, performance metrics are desirable. > This proposes adding them on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org