[
https://issues.apache.org/jira/browse/HADOOP-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16466106#comment-16466106
]
Daryn Sharp commented on HADOOP-13738:
--------------------------------------
This jira was not thought out. It's causing problems on the clusters where
it's deployed.
# We had a cluster lose 10% of nodes due to this patch. A few nodes filled up,
they went dead, and created a domino effect that caused nodes to go dead until
intervention.
# Jobs may cause severe performance degradation from sync storms during a sort
phase because local dir allocator calls checkDisk.
The biggest risk is a runaway job may fill disks and cause a cluster to implode.
[~arpitagarwal], do you want me to file another jira for immediate revert? Or
do you want to reopen this one?
> DiskChecker should perform some disk IO
> ---------------------------------------
>
> Key: HADOOP-13738
> URL: https://issues.apache.org/jira/browse/HADOOP-13738
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Priority: Major
> Fix For: 2.9.0, 3.0.0-alpha2, 2.8.4
>
> Attachments: HADOOP-13738-branch-2.8-06.patch, HADOOP-13738.01.patch,
> HADOOP-13738.02.patch, HADOOP-13738.03.patch, HADOOP-13738.04.patch,
> HADOOP-13738.05.patch
>
>
> DiskChecker can fail to detect total disk/controller failures indefinitely.
> We have seen this in real clusters. DiskChecker performs simple
> permissions-based checks on directories which do not guarantee that any disk
> IO will be attempted.
> A simple improvement is to write some data and flush it to the disk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
