[
https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985134#comment-15985134
]
Erik Krogen edited comment on HDFS-8873 at 4/26/17 5:05 PM:
------------------------------------------------------------
I've noticed the new {{TestDirectoryScanner#testThrottling}} fails consistently
on OS X... I filed HDFS-11707 to track.
Also attaching a patch for a 2.7 backport; it does not contain the findbugs
issue fixed in HDFS-9174. The backport was mostly clean with some minor
non-structural conflicts around {{DirectoryScanner#getDiskReport}} caused by
HDFS-7758 "Retire FsDatasetSpi#getVolumes() and use
FsDatasetSpi#getVolumeRefs()" which moved the order of a few statements within
{{getDiskReport}}. That change doesn't have an actual effect on the behavior of
this patch.
was (Author: xkrogen):
I've noticed the new {{TestDirectoryScanner#testThrottling}} fails consistently
on OS X... I filed HDFS-11707 to track.
Also attaching a patch for a 2.7 backport; it does not contain the findbugs
issue fixed in HDFS-9174.
> Allow the directoryScanner to be rate-limited
> ---------------------------------------------
>
> Key: HDFS-8873
> URL: https://issues.apache.org/jira/browse/HDFS-8873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Nathan Roberts
> Assignee: Daniel Templeton
> Labels: 2.7.2-candidate
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch,
> HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch,
> HDFS-8873.006.patch, HDFS-8873.007.patch, HDFS-8873.008.patch,
> HDFS-8873.009.patch, HDFS-8873-branch-2.7.009.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms
> of disk seeks (see HDFS-8791) for details.
> It would be good if the directoryScanner() had a configurable duty cycle that
> would reduce its impact on disk performance (much like the approach in
> HDFS-8617).
> Without such a throttle, disks can go 100% busy for many minutes at a time
> (assuming the common case of all inodes in cache but no directory blocks
> cached, 64K seeks are required for full directory listing which translates to
> 655 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
