[ 
https://issues.apache.org/jira/browse/HDFS-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852294#comment-17852294
 ] 

ZanderXu commented on HDFS-17366:
---------------------------------

Thanks [~zhengchenyu] for your watching. 
{quote}Do we need to acquire the inode file lock for block every time we 
process a block? If so, will this make the block report slower?
{quote}
Yes, BR/IBR should acquire the INodeFileLock to process a block. About the 
performance, BR/IBR may take more time, but if we take the waiting time into 
account, the total time of BR/IBR may reduced, especially for the high load 
namenode. 

 
{quote}For deleting the block lock, we first obtain the inode file lock, and 
then obtain the dn lock
{quote}
Block deleting logic just acquire the INodeFileLock, will not acquire the DN 
Lock.  HDFS-13671  introduced relationship between blocks, it caused that 
BR/IBR acquire multiple INodeFileLocks when processing one block. So the 
HDFS-13671 should be reverted, and we can find a new solution to fix this 
performance problem.

 

> NameNode Fine-Grained Locking via Namespace Tree
> ------------------------------------------------
>
>                 Key: HDFS-17366
>                 URL: https://issues.apache.org/jira/browse/HDFS-17366
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs, namenode
>            Reporter: ZanderXu
>            Assignee: ZanderXu
>            Priority: Major
>         Attachments: NameNode Fine-Grained Locking Based On Directory Tree.pdf
>
>
> As we all known, the write performance of NameNode is limited by the global 
> lock. We target to enable fine-grained locking based on the Namespace tree to 
> improve the performance of NameNode write operations.
> There are multiple motivations for creating this ticket:
>  * We have implemented this fine-grained locking and gained nearly 7x 
> performance improvements in our prod environment
>  * Other companies made similar improvements based on their internal branch. 
> Internal branches are quite different from the community, so few feedback and 
> discussions in the community.
>  * The topic of fine-grained locking has been discussed for a very long time, 
> but still without any results.
>  
> We implemented this fine-gained locking based on the namespace tree to 
> maximize the number of concurrency for disjoint or independent operations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to