[
https://issues.apache.org/jira/browse/HDFS-17098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829314#comment-17829314
]
ASF GitHub Bot commented on HDFS-17098:
---------------------------------------
teamconfx commented on code in PR #6035:
URL: https://github.com/apache/hadoop/pull/6035#discussion_r1532946326
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##########
@@ -666,7 +666,15 @@ private Consumer<List<DatanodeInfoWithStorage>>
createSecondaryNodeSorter() {
Consumer<List<DatanodeInfoWithStorage>> secondarySort = null;
if (readConsiderStorageType) {
Comparator<DatanodeInfoWithStorage> comp =
- Comparator.comparing(DatanodeInfoWithStorage::getStorageType);
+ Comparator.comparing(DatanodeInfoWithStorage::getStorageType, (s1,
s2) -> {
+ if (s1 == null) {
Review Comment:
@Hexiaoqiao we got this when we set the following configuration
"dfs.heartbeat.interval=1753310367" and
"dfs.namenode.read.considerStorageType="true". Under this config, the test
"org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager#testNoLogs"
would trigger the case.
> DatanodeManager does not handle null storage type properly
> ----------------------------------------------------------
>
> Key: HDFS-17098
> URL: https://issues.apache.org/jira/browse/HDFS-17098
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: ConfX
> Priority: Critical
> Labels: pull-request-available
> Attachments: reproduce.sh
>
>
> h2. What happened:
> Got a {{NullPointerException}} without message when sorting datanodes in
> {{{}NetworkTopology{}}}.
> h2. Where's the bug:
> In line 654 of {{{}DatanodeManager{}}}, the manager creates a second sorter
> using the standard {{Comparator}} class:
> {noformat}
> Comparator<DatanodeInfoWithStorage> comp =
> Comparator.comparing(DatanodeInfoWithStorage::getStorageType);
> secondarySort = list -> Collections.sort(list, comp);{noformat}
> This comparator is then used in {{NetworkTopology}} as a secondary sort to
> break ties:
> {noformat}
> if (secondarySort != null) {
> // a secondary sort breaks the tie between nodes.
> secondarySort.accept(nodesList);
> }{noformat}
> However, if the storage type is {{{}null{}}}, a {{NullPointerException}}
> would be thrown since the default {{Comparator.comparing}} cannot handle
> comparison between null values.
> h2. How to reproduce:
> (1) Set {{dfs.heartbeat.interval}} to {{{}1753310367{}}}, and
> {{dfs.namenode.read.considerStorageType}} to {{true}}
> (2) Run test:
> {{org.apache.hadoop.hdfs.server.blockmanagement.TestSortLocatedBlock#testAviodStaleAndSlowDatanodes}}
> h2. Stacktrace:
> {noformat}
> java.lang.NullPointerException
> at
> java.base/java.util.Comparator.lambda$comparing$77a9974f$1(Comparator.java:469)
> at java.base/java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.base/java.util.TimSort.sort(TimSort.java:220)
> at java.base/java.util.Arrays.sort(Arrays.java:1515)
> at java.base/java.util.ArrayList.sort(ArrayList.java:1750)
> at java.base/java.util.Collections.sort(Collections.java:179)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.lambda$createSecondaryNodeSorter$0(DatanodeManager.java:654)
> at
> org.apache.hadoop.net.NetworkTopology.sortByDistance(NetworkTopology.java:983)
> at
> org.apache.hadoop.net.NetworkTopology.sortByDistanceUsingNetworkLocation(NetworkTopology.java:946)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlock(DatanodeManager.java:637)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlocks(DatanodeManager.java:554)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestSortLocatedBlock.testAviodStaleAndSlowDatanodes(TestSortLocatedBlock.java:144){noformat}
> For an easy reproduction, run the reproduce.sh in the attachment. We are
> happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
