[
https://issues.apache.org/jira/browse/HDFS-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596712#comment-14596712
]
Haohui Mai commented on HDFS-8617:
----------------------------------
bq. You can read my related SoCC paper here:
http://umbrant.com/papers/socc12-cake.pdf . I experimented with ioprio about 3
years ago as part of this work, and didn't get positive results. We needed
application-level throttling.
As you mentioned in the evaluation, there are adverse effects on throughputs.
I agree that application-level throttling can be useful. The proposed solution,
however, relies on magic numbers to work. My concern is that how to choose the
magic numbers? Is it repeatable to achieve good performance? Is it
generalizable to other configuration? It looks to me that currently the answers
of both questions are no. The proposed solution looks like lowering the
utilization of the cluster (at the cost of making {{checkDir()}} really slow)
to meet the SLOs.
bq. The key issue though, as both Colin and I have mentioned, is that there is
queuing both in the OS and on disk. ioprio only affects OS-level queuing, and
disk-level queuing can be quite substantial. Not sure how much more needs to be
said.
Point taken. Unfortunately without performance benchmarks and numbers the
statements are purely speculative. For example, what do you mean by
substantial? The size of the NCQ is 32 compared the size of OS level I/O queue
can be hundreds or thousands. I really appreciate doing some performance
benchmarks and sharing the numbers.
My concern of the proposal is that the parameter cannot be automatically
tunable w.r.t. cluster configurations and loads. It has to be dynamic. In the
longer term it makes a lot sense to tune these parameters based on the length
of the I/O queue, avg. processing time, etc. At the first step I think it can
be very helpful to simply correlate these parameters with simple metrics like
the number of tranceiver threads.
> Throttle DiskChecker#checkDirs() speed.
> ---------------------------------------
>
> Key: HDFS-8617
> URL: https://issues.apache.org/jira/browse/HDFS-8617
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
> Affects Versions: 2.7.0
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Attachments: HDFS-8617.000.patch
>
>
> As described in HDFS-8564, {{DiskChecker.checkDirs(finalizedDir)}} is
> causing excessive I/Os because {{finalizedDirs}} might have up to 64K
> sub-directories (HDFS-6482).
> This patch proposes to limit the rate of IO operations in
> {{DiskChecker.checkDirs()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
