[
https://issues.apache.org/jira/browse/HDFS-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14592685#comment-14592685
]
Andrew Wang commented on HDFS-8617:
-----------------------------------
In this case, we've got a single DiskChecker thread, presumably with ~1
outstanding IO. This little amount of extra queuing from DiskChecker is still
impacting HBase a lot. We're not talking 1000s of concurrent IOs here, which
seems like a lot in any case since the default nr_requests is 128.
Maybe Esteban can provide some additional OS-level metrics, it could be that
other effects like extra context switches are also hurting performance.
> Throttle DiskChecker#checkDirs() speed.
> ---------------------------------------
>
> Key: HDFS-8617
> URL: https://issues.apache.org/jira/browse/HDFS-8617
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
> Affects Versions: 2.7.0
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Attachments: HDFS-8617.000.patch
>
>
> As described in HDFS-8564, {{DiskChecker.checkDirs(finalizedDir)}} is
> causing excessive I/Os because {{finalizedDirs}} might have up to 64K
> sub-directories (HDFS-6482).
> This patch proposes to limit the rate of IO operations in
> {{DiskChecker.checkDirs()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
