[
https://issues.apache.org/jira/browse/HDFS-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596748#comment-14596748
]
Andrew Wang commented on HDFS-8617:
-----------------------------------
bq. As you mentioned in the evaluation, there are adverse effects on
throughputs...The proposed solution looks like lowering the utilization of the
cluster (at the cost of making checkDir() really slow) to meet the SLOs.
I'd like to turn this question around and ask: is there a downside to
throttling checkDisk throughput? We might end up taking longer to detect a bad
disk, but this is not a performance-critical workload.
Here's also another idea for a throttle: spend at most x% of time doing
checkDisk work. Maybe we say it can only run for 250ms of every 1000ms
interval. Timeslicing like this automatically tunes for faster vs. slower IO
rates.
> Throttle DiskChecker#checkDirs() speed.
> ---------------------------------------
>
> Key: HDFS-8617
> URL: https://issues.apache.org/jira/browse/HDFS-8617
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
> Affects Versions: 2.7.0
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Attachments: HDFS-8617.000.patch
>
>
> As described in HDFS-8564, {{DiskChecker.checkDirs(finalizedDir)}} is
> causing excessive I/Os because {{finalizedDirs}} might have up to 64K
> sub-directories (HDFS-6482).
> This patch proposes to limit the rate of IO operations in
> {{DiskChecker.checkDirs()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
