[
https://issues.apache.org/jira/browse/HDFS-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888830#comment-15888830
]
Arpit Agarwal edited comment on HDFS-11461 at 2/28/17 8:41 PM:
---------------------------------------------------------------
Thanks [~hanishakoneru]. The patch looks good. A few more comments:
# diskOutliersStats should be updated atomically. Else if someone reads the map
after the {{clear}} call but before the for loop completes they will see
partial results. You can allocate a new map in {{detectAndUpdateDiskOutliers}}
and replace the reference which will be atomic.
# We can move startDiskOutlierDetectionThread, detectAndUpdateDiskOutliers and
diskOutliersStats to a new package-private class. Also the shutdown logic can
be moved to a method of the new class. e.g. shutdownAndWait. Sorry I should
have suggested this before.
# diskOutliersStats can be defined as a Map<String, Map<OpType, Double>>, so
that it's explicit which values are read/write/meta.
Also once the logic is isolated to a new class, see if you can write a unit
test (mock-based may be easiest). Ok if you want to file a separate Jira to do
that.
was (Author: arpitagarwal):
Thanks [~hanishakoneru]. The patch looks good. A few more comments:
# diskOutliersStats should be updated atomically. Else if someone reads the map
after the {{clear}} call but before the for loop completes they will see
partial results. You can allocate a new map in {{detectAndUpdateDiskOutliers}}
and replace the reference which will be atomic.
# We can move startDiskOutlierDetectionThread, detectAndUpdateDiskOutliers and
diskOutliersStats to a new package-private class. Also the shutdown logic can
be moved to a method of the new class. e.g. shutdownAndWait. Sorry I should
have suggested this before.
# diskOutliersStats can be defined as a Map<String, Map<OpType, Double>>, so
that its explicit which values are read/write/meta.
Also once the logic is isolated to a new class, see if you can write a unit
test (mock-based may be easiest). Ok if you want to file a separate Jira to do
that.
> DataNode Disk Outlier Detection
> -------------------------------
>
> Key: HDFS-11461
> URL: https://issues.apache.org/jira/browse/HDFS-11461
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Reporter: Hanisha Koneru
> Assignee: Hanisha Koneru
> Attachments: HDFS-11461.000.patch, HDFS-11461.001.patch
>
>
> Similar to how DataNodes collect peer performance statistics, we can collect
> disk performance statistics per datanode and detect outliers among them, if
> any.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
