[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011549#comment-17011549
]
Vinayakumar B edited comment on HDFS-14578 at 1/9/20 8:36 AM:
--------------------------------------------------------------
Code change looks good.
some improvements required in tests.
Right now, distribution of nodes capacity makes one entire rack to be 75% full
and another rack to be empty.
This will result in choosing localrack node with same usage (75%) as of local
node in {{testChooseLocalNodeWihLocalNodeLoaded()}}
Distribute the datanodes evenly in both racks, (i.e. Some are full and some are
empty in both racks so that there would be a better rack-local node to choose
when local node is full)
Tests can be made simple by asserting expected node to be choosen instead
calculating the probability.
1. {{testChooseLocalNode()}} assert for local node.
2. {{testChooseLocalNodeWihLocalNodeLoaded()}}, assert for non-local, but
rack-local with higher space availability than local node.
was (Author: vinayrpet):
Code change looks good.
some improvements required in tests.
Right now, distribution of nodes capacity makes one entire rack to be 75% full
and another rack to be empty.
This will result in choosing localrack node with same usage (75%) as of local
node in {{testChooseLocalNodeWihLocalNodeLoaded()}}
Distribute the datanodes evenly in both racks, (i.e. Some are full and some are
empty in both racks so that there would be a better rack-local node to choose
when local node is full)
Tests can be made simple by asserting expected node to be choosen instead
calculating the probability.
1. {{testChooseLocalNode()}} assert for local node.
1. {{testChooseLocalNodeWihLocalNodeLoaded()}}, assert for non-local, but
rack-local with higher space availability than local node.
> AvailableSpaceBlockPlacementPolicy always prefers local node
> ------------------------------------------------------------
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
> Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
> Reporter: Wei-Chiu Chuang
> Assignee: Ayush Saxena
> Priority: Major
> Attachments: HDFS-14578-02.patch, HDFS-14578-03.patch,
> HDFS-14578-04.patch, HDFS-14578-WIP-01.patch, HDFS-14758-01.patch
>
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
