[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871239#comment-16871239
]
Ayush Saxena commented on HDFS-14578:
-------------------------------------
Thanx for confirming, We didn't got such complain, so I never analyzed much,
but out of curiosity I tried something, a 5 min code(Just to confirm, if this
is the correct direction towards solution)
We can put something like this in {{AvailableSpaceBlockPlacementPolicy}}
{code:java}
@Override
protected DatanodeStorageInfo chooseLocalStorage(Node localMachine,
Set<Node> excludedNodes, long blocksize, int maxNodesPerRack,
List<DatanodeStorageInfo> results, boolean avoidStaleNodes,
EnumMap<StorageType, Integer> storageTypes, boolean fallbackToLocalRack)
throws NotEnoughReplicasException {
if (!optimizeLocal) {
return super.chooseLocalStorage(localMachine, excludedNodes, blocksize,
maxNodesPerRack, results, avoidStaleNodes, storageTypes,
fallbackToLocalRack);
}
final EnumMap<StorageType, Integer> initialStorageTypesLocal =
storageTypes.clone();
final EnumMap<StorageType, Integer> initialStorageTypesLocalRack =
storageTypes.clone();
DatanodeStorageInfo local = chooseLocalStorage(localMachine, excludedNodes,
blocksize, maxNodesPerRack, results, avoidStaleNodes,
initialStorageTypesLocal);
if (!fallbackToLocalRack) {
return local;
}
DatanodeStorageInfo localRack =
chooseLocalRack(localMachine, excludedNodes, blocksize, maxNodesPerRack,
results, avoidStaleNodes, initialStorageTypesLocalRack);
if (local != null && localRack != null) {
if (select(local.getDatanodeDescriptor(),
localRack.getDatanodeDescriptor()) == local.getDatanodeDescriptor()) {
results.remove(localRack);
storageTypes = initialStorageTypesLocal;
return local;
} else {
results.remove(local);
storageTypes = initialStorageTypesLocalRack;
return localRack;
}
} else if (localRack == null) {
storageTypes = initialStorageTypesLocal;
return local;
} else {
storageTypes = initialStorageTypesLocalRack;
return localRack;
}
}{code}
Based on some config, we can have a logic like this, else follow the old trend.
I know its very raw code, just to catch up idea, if something like this is
intended. We may think a better way to do it. Do let me know if something else
is intended. I will try to give a try to that too.:)
> AvailableSpaceBlockPlacementPolicy always prefers local node
> ------------------------------------------------------------
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
> Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
