[ https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011549#comment-17011549 ]
Vinayakumar B edited comment on HDFS-14578 at 1/9/20 8:36 AM: -------------------------------------------------------------- Code change looks good. some improvements required in tests. Right now, distribution of nodes capacity makes one entire rack to be 75% full and another rack to be empty. This will result in choosing localrack node with same usage (75%) as of local node in {{testChooseLocalNodeWihLocalNodeLoaded()}} Distribute the datanodes evenly in both racks, (i.e. Some are full and some are empty in both racks so that there would be a better rack-local node to choose when local node is full) Tests can be made simple by asserting expected node to be choosen instead calculating the probability. 1. {{testChooseLocalNode()}} assert for local node. 2. {{testChooseLocalNodeWihLocalNodeLoaded()}}, assert for non-local, but rack-local with higher space availability than local node. was (Author: vinayrpet): Code change looks good. some improvements required in tests. Right now, distribution of nodes capacity makes one entire rack to be 75% full and another rack to be empty. This will result in choosing localrack node with same usage (75%) as of local node in {{testChooseLocalNodeWihLocalNodeLoaded()}} Distribute the datanodes evenly in both racks, (i.e. Some are full and some are empty in both racks so that there would be a better rack-local node to choose when local node is full) Tests can be made simple by asserting expected node to be choosen instead calculating the probability. 1. {{testChooseLocalNode()}} assert for local node. 1. {{testChooseLocalNodeWihLocalNodeLoaded()}}, assert for non-local, but rack-local with higher space availability than local node. > AvailableSpaceBlockPlacementPolicy always prefers local node > ------------------------------------------------------------ > > Key: HDFS-14578 > URL: https://issues.apache.org/jira/browse/HDFS-14578 > Project: Hadoop HDFS > Issue Type: Bug > Components: block placement > Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1 > Reporter: Wei-Chiu Chuang > Assignee: Ayush Saxena > Priority: Major > Attachments: HDFS-14578-02.patch, HDFS-14578-03.patch, > HDFS-14578-04.patch, HDFS-14578-WIP-01.patch, HDFS-14758-01.patch > > > It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like > in the BlockPlacementPolicyDefault > > As Yongjun mentioned in > [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739], > > {quote}Class AvailableSpaceBlockPlacementPolicy extends > BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing > the first node in BlockPlacementPolicyDefault, so even with this new feature, > the local DN is always chosen as the first DN (of course when it is not > excluded), and the new feature only changes the selection of the rest of the > two DNs. > {quote} > I'm file this Jira as I groom Cloudera's internal Jira and found this > unreported issue. We do have a customer hitting this problem. I don't have a > fix, but thought it would be beneficial to report it to Apache Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org