[jira] [Work logged] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

Tue, 08 Feb 2022 23:39:05 -0800 


     [ 
https://issues.apache.org/jira/browse/HDFS-16432?focusedWorklogId=723479&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-723479
 ]

ASF GitHub Bot logged work on HDFS-16432:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 09/Feb/22 07:38
            Start Date: 09/Feb/22 07:38
    Worklog Time Spent: 10m 
      Work Description: liubingxing edited a comment on pull request #3907:
URL: https://github.com/apache/hadoop/pull/3907#issuecomment-1033432213


    Hi @sodonnel, I have tried reproducing this problem on a test cluster and 
the cpu profile of block report process look like as follow.
   
![image](https://user-images.githubusercontent.com/2844826/153139265-e0780e60-c924-4720-8680-16f1dfebac3c.png)
   
![image](https://user-images.githubusercontent.com/2844826/153139376-f57b4ab4-e153-4ca3-819c-95afb49e313f.png)
   
![image](https://user-images.githubusercontent.com/2844826/153139426-48d27252-4367-4e9f-9457-1e128789a0b1.png)
   
   It looks like most of the time were taking by the #processReportedBlock and 
#moveBlockToHead, and the #moveBlockToHead is necessary after removing the 
folded tree set structure.
   If you have any good suggestions, please let me know.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


Issue Time Tracking
-------------------

    Worklog Id:     (was: 723479)
    Time Spent: 2h  (was: 1h 50m)

> Namenode block report add yield to avoid holding write lock too long
> --------------------------------------------------------------------
>
>                 Key: HDFS-16432
>                 URL: https://issues.apache.org/jira/browse/HDFS-16432
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: qinyuren
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: image-2022-01-20-15-19-28-384.png
>
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> !image-2022-01-20-15-19-28-384.png|width=683,height=132!
> In our cluster, namenode block report will held write lock for a long time if 
> the storage block number more than 100000. So we want to add a yield 
> mechanism in block reporting process to avoid holding write lock too long.
>  # Ensure that the processing of the same block is in the same write lock.
>  # Because StorageInfo.addBlock will moves the block to the head of 
> blockList, so we can collect blocks that have not been reported by delimiter 
> block.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to