[
https://issues.apache.org/jira/browse/HDFS-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076284#comment-17076284
]
Stephen O'Donnell commented on HDFS-15255:
------------------------------------------
I'd like to confirm I understand how this works correctly.
The nodes are first sorted by "distance from the client". There is then a
secondary sort, which by default is a shuffle which is applied to nodes with
the same distance.
This change will allow the "same distance nodes" to be sorted by storage type,
putting the fastest storage first. We also have an option to sort by DN load,
added by HDFS-14882.
Looking at the code, it seems this new change (sort by storage type) is not
compatible with "sort by load" (HDFS-14882). Is that correct? If both are set,
then the comparator here would return just the last one:
{code}
private Consumer<List<DatanodeInfoWithStorage>> createSecondaryNodeSorter() {
Consumer<List<DatanodeInfoWithStorage>> secondarySort =
list -> Collections.shuffle(list);
if (readConsiderLoad) {
Comparator<DatanodeInfoWithStorage> comp =
Comparator.comparingInt(DatanodeInfo::getXceiverCount);
secondarySort = list -> Collections.sort(list, comp);
}
if (readConsiderStorageType) {
Comparator<DatanodeInfoWithStorage> comp =
Comparator.comparing(DatanodeInfoWithStorage::getStorageType);
secondarySort = list -> Collections.sort(list, comp);
}
return secondarySort;
}
{code}
1. We would need to document this limitation in the hdfs-site.xml for both
`dfs.namenode.read.consider.storageType` and `dfs.namenode.read.considerLoad`
to make that clear.
2. We should probably also align the two parameters so they have similar names
for consistency, eg change this one to `dfs.namenode.read.considerStorageType`.
3. Could you add a test for this change please - you could probably use the one
in HDFS-14882 as a starting point.
4. Just a question - if network distance is always the same, and there is only
1 SSD storage and this new feature is enabled. Would the SSD storage be the
only one ever picked? Could like result in load problems on that host?
> Consider StorageType when DatanodeManager#sortLocatedBlock()
> ------------------------------------------------------------
>
> Key: HDFS-15255
> URL: https://issues.apache.org/jira/browse/HDFS-15255
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Lisheng Sun
> Assignee: Lisheng Sun
> Priority: Major
> Attachments: HDFS-15255.001.patch
>
>
> When only one replica of a block is SDD, the others are HDD.
> When the client reads the data, the current logic is that it considers the
> distance between the client and the dn. I think it should also consider the
> StorageType of the replica. Priority to return a replica of the specified
> StorageType
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
