[
https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219581#comment-15219581
]
Nicolas Fraison commented on HDFS-10220:
----------------------------------------
[~raviprak] this issue seems to only happens when we have for different reason
a big MR job or a big number of map-reduce jobs abnormally failing (nso not
well closing lease).
It is quite difficult to tll you number of open files at there are lots of
different jobs running on the platform (up to 100K per days) with different
behaviour. Lot's of them are distcp jobs with a max of 300 files open for write
by each distcp. So we should have a maximum of 20-40K files open for write at a
time.
??Do you think we should cycle through all the leases rather than the same set
every iteration? We may be over-engineering, but I'd be interested in your
opinion??
In fact the patch is quite simple and will cycle on the same set. I think that
it will be better to cycle through all expired lease on multiple iteration.
Unless we consider that checking for 100k lease is a safe value and that we
should not cycle on the same set as some will be released.
??Please rename nPathReleaseExpiredLease to numLeasesReleased??
I will rename it numLeasesPathsReleased as it is the number of paths treated in
the loop releasing lease.
> Namenode failover due to too long loking in LeaseManager.Monitor
> ----------------------------------------------------------------
>
> Key: HDFS-10220
> URL: https://issues.apache.org/jira/browse/HDFS-10220
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Reporter: Nicolas Fraison
> Priority: Minor
> Attachments: HADOOP-10220.001.patch, threaddump_zkfc.txt
>
>
> I have faced a namenode failover due to unresponsive namenode detected by the
> zkfc with lot's of WARN messages (5 millions) like this one:
> _org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease: All
> existing blocks are COMPLETE, lease removed, file closed._
> On the threaddump taken by the zkfc there are lots of thread blocked due to a
> lock.
> Looking at the code, there are a lock taken by the LeaseManager.Monitor when
> some lease must be released. Due to the really big number of lease to be
> released the namenode has taken too many times to release them blocking all
> other tasks and making the zkfc thinking that the namenode was not
> available/stuck.
> The idea of this patch is to limit the number of leased released each time we
> check for lease so the lock won't be taken for a too long time period.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
