[
https://issues.apache.org/jira/browse/HDFS-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944327#comment-14944327
]
Uma Maheswara Rao G commented on HDFS-9137:
-------------------------------------------
Thanks for the comments, Yi , Vinay and Colin. Here actually me and Yi
discussed initially to cleanup toString and later I found triggerBlockReport
itself unnecessary inside lock and me and Yi concluded we work on toString
separately. So removing whole BR under lock can fix this deadlock. I just filed
HDFS-9200 for tracking toString issue.
Coming to comments, Yes I was trying to avoid suppressing root exceptions but
it turning to have more if check. Let me throw directly then. Also above patch
missed to copy the triggerBR exception, which will miss to throw exception out
if there is any exception only in BR.
> DeadLock between DataNode#refreshVolumes and
> BPOfferService#registrationSucceeded
> ----------------------------------------------------------------------------------
>
> Key: HDFS-9137
> URL: https://issues.apache.org/jira/browse/HDFS-9137
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.0.0, 2.7.1
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-9137.00.patch
>
>
> I can see this code flows between DataNode#refreshVolumes and
> BPOfferService#registrationSucceeded could cause deadLock.
> In practice situation may be rare as user calling refreshVolumes at the time
> DN registration with NN. But seems like issue can happen.
> Reason for deadLock:
> 1) refreshVolumes will be called with DN lock and after at the end it will
> also trigger Block report. In the Block report call,
> BPServiceActor#triggerBlockReport calls toString on bpos. Here it takes
> readLock on bpos.
> DN lock then boos lock
> 2) BPOfferSetrvice#registrationSucceeded call is taking writeLock on bpos and
> calling dn.bpRegistrationSucceeded which is again synchronized call on DN.
> bpos lock and then DN lock.
> So, this can clearly create dead lock.
> I think simple fix could be to move triggerBlockReport call outside out DN
> lock and I feel that call may not be really needed inside DN lock.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
