[
https://issues.apache.org/jira/browse/HDFS-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6961:
----------------------------
Attachment: HDFS-6961.002.patch
Thanks for the comments, Nicholas and Vinay! Update the patch to address the
comments.
bq. "if (!unavailableStorages.contains(type))" is not needed. "boolean retry"
is also not needed since storageTypes must be non-empty
Actually we may need the retry check here. This is because in {{chooseTarget}},
if the {{storageTypes}} is empty, another {{NotEnoughReplicasException}} will
be thrown and we can hit an infinite loop without the check in
{{unavailableStorages}} (the infinite loop can be produced by
TestReplicationPolicy#testChooseTargetWithMoreThanAvailableNodesWithStaleness):
{code}
try {
if ((numOfReplicas = requiredStorageTypes.size()) == 0) {
throw new NotEnoughReplicasException(
"All required storage types are unavailable: "
+ " unavailableStorages=" + unavailableStorages
+ ", storagePolicy=" + storagePolicy);
}
if (numOfResults == 0) {
{code}
bq. Then above iterator may return HDD storageType first, then LocalStorage
will be chosen with HDD as storage type instead of SSD.
I think you are right here, Vinay. For Archival Storage, what we can do here
may be switching the two for loops, since EnumMap.entrySet().iterator keeps the
order of the key, and DISK is before ARCHIVE in StorageType. To completely
solve the storage preference issue we may need better mechanism and maybe we
can do it in a separate jira?
> Archival Storage: BlockPlacementPolicy#chooseTarget should check each valid
> storage type in each choosing round
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-6961
> URL: https://issues.apache.org/jira/browse/HDFS-6961
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer, namenode
> Reporter: Jing Zhao
> Assignee: Jing Zhao
> Attachments: HDFS-6961.000.patch, HDFS-6961.001.patch,
> HDFS-6961.002.patch
>
>
> Currently in each choosing round of BlockPlacementPolicy#chooseTarget (e.g.,
> round 1 for choosing local storage, round 2 for remote rack, etc.), we only
> take into account the first storage type in the storage type list. This may
> fail to choose the local machine or sometimes may even fail to choose enough
> nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
