[
https://issues.apache.org/jira/browse/HDFS-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596627#comment-14596627
]
Andrew Wang commented on HDFS-8617:
-----------------------------------
You can read my related SoCC paper here:
http://umbrant.com/papers/socc12-cake.pdf . I experimented with ioprio about 3
years ago as part of this work, and didn't get positive results. We needed
application-level throttling.
The key issue though, as both Colin and I have mentioned, is that there is
queuing both in the OS and on disk. ioprio only affects OS-level queuing, and
disk-level queuing can be quite substantial. Not sure how much more needs to be
said.
Also as Colin (and I) mentioned, deadline and noop IO schedulers are often used
for latency sensitive workloads like HBase, and ioprio only works with CFQ.
Thus ioprio is not going to work in this situation.
> Throttle DiskChecker#checkDirs() speed.
> ---------------------------------------
>
> Key: HDFS-8617
> URL: https://issues.apache.org/jira/browse/HDFS-8617
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
> Affects Versions: 2.7.0
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Attachments: HDFS-8617.000.patch
>
>
> As described in HDFS-8564, {{DiskChecker.checkDirs(finalizedDir)}} is
> causing excessive I/Os because {{finalizedDirs}} might have up to 64K
> sub-directories (HDFS-6482).
> This patch proposes to limit the rate of IO operations in
> {{DiskChecker.checkDirs()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
