[
https://issues.apache.org/jira/browse/HDFS-16540?focusedWorklogId=759978&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-759978
]
ASF GitHub Bot logged work on HDFS-16540:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Apr/22 12:26
Start Date: 21/Apr/22 12:26
Worklog Time Spent: 10m
Work Description: ndimiduk commented on code in PR #4170:
URL: https://github.com/apache/hadoop/pull/4170#discussion_r855124598
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##########
@@ -1189,16 +1190,25 @@ public void registerDatanode(DatanodeRegistration
nodeReg)
nodes with its data cleared (or user can just remove the StorageID
value in "VERSION" file under the data directory of the datanode,
but this is might not work if VERSION file format has changed
- */
+ */
+ // Check if nodeS's host information is same as nodeReg's, if not,
+ // it needs to update host2DatanodeMap accordringly.
+ updateHost2DatanodeMap =
!nodeS.getXferAddr().equals(nodeReg.getXferAddr());
+
NameNode.stateChangeLog.info("BLOCK* registerDatanode: " + nodeS
+ " is replaced by " + nodeReg + " with the same storageID "
- + nodeReg.getDatanodeUuid());
+ + nodeReg.getDatanodeUuid() + ", updateHost2DatanodeMap: " +
updateHost2DatanodeMap);
Review Comment:
Is this extra information needed at the `INFO` level log? I understand that
having the value printed is helpful during development, but I don't think it's
meaningful to an operator.
Also, if you're here to change a log message, maybe also change it to use
the format string version instead of string concatenation?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##########
@@ -1189,16 +1190,25 @@ public void registerDatanode(DatanodeRegistration
nodeReg)
nodes with its data cleared (or user can just remove the StorageID
value in "VERSION" file under the data directory of the datanode,
but this is might not work if VERSION file format has changed
- */
+ */
+ // Check if nodeS's host information is same as nodeReg's, if not,
+ // it needs to update host2DatanodeMap accordringly.
+ updateHost2DatanodeMap =
!nodeS.getXferAddr().equals(nodeReg.getXferAddr());
Review Comment:
Can `nodeS.getXferAddr()` ever be null? Use of `Objects.equals()` would be
safer.
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java:
##########
@@ -138,6 +138,35 @@ public void testNumVersionsCorrectAfterReregister()
mapToCheck.get("version1").intValue(), 1);
}
+ /**
+ * This test checks that if a node is re-registered with a different ip, its
+ * host2DatanodeMap is correctly updated with the new ip.
+ */
+ @Test
Review Comment:
It looks like there's a couple caches like this one. Is there equivalent
test coverage for all the registered DN state that is updated on registration?
Can you add coverage for any of the other states that are missing?
Issue Time Tracking
-------------------
Worklog Id: (was: 759978)
Time Spent: 1.5h (was: 1h 20m)
> Data locality is lost when DataNode pod restarts in kubernetes
> ---------------------------------------------------------------
>
> Key: HDFS-16540
> URL: https://issues.apache.org/jira/browse/HDFS-16540
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.3.2
> Reporter: Huaxiang Sun
> Assignee: Huaxiang Sun
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod
> restarts, we found that data locality is lost after we do a major compaction
> of hbase regions. After some debugging, we found that upon pod restarts, its
> ip changes. In DatanodeManager, maps like networktopology are updated with
> the new info. host2DatanodeMap is not updated accordingly. When hdfs client
> with the new ip tries to find a local DataNode, it fails.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
