[ 
https://issues.apache.org/jira/browse/HDFS-9412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-9412:
----------------------------------
    Release Note: Skip blocks with size below 
dfs.balancer.getBlocks.min-block-size (default 10MB) when a balancer asks for a 
list of blocks.

> getBlocks occupies FSLock and takes too long to complete
> --------------------------------------------------------
>
>                 Key: HDFS-9412
>                 URL: https://issues.apache.org/jira/browse/HDFS-9412
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover, namenode
>            Reporter: He Tianyi
>            Assignee: He Tianyi
>            Priority: Major
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
>         Attachments: HDFS-9412-branch-2.7.00.patch, HDFS-9412.0000.patch, 
> HDFS-9412.0001.patch, HDFS-9412.0002.patch
>
>
> {{getBlocks}} in {{NameNodeRpcServer}} acquires a read lock then may take a 
> long time to complete (probably several seconds, if number of blocks are too 
> much). 
> During this period, other threads attempting to acquire write lock will wait. 
> In an extreme case, RPC handlers are occupied by one reader thread calling 
> {{getBlocks}} and all other threads waiting for write lock, rpc server acts 
> like hung. Unfortunately, this tends to happen in heavy loaded cluster, since 
> read operations come and go fast (they do not need to wait), leaving write 
> operations waiting.
> Looks like we can optimize this thing like DN block report did in past, by 
> splitting the operation into smaller sub operations, and let other threads do 
> their work between each sub operation. The whole result is returned at once, 
> though (one thing different from DN block report). 
> I am not sure whether this will work. Any better idea?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to