Hexiaoqiao commented on code in PR #5398:
URL: https://github.com/apache/hadoop/pull/5398#discussion_r1108243429
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/IncrementalBlockReportManager.java:
##########
@@ -252,7 +256,9 @@ synchronized void addRDBI(ReceivedDeletedBlockInfo rdbi,
// Make sure another entry for the same block is first removed.
// There may only be one such entry.
for (PerStorageIBR perStorage : pendingIBRs.values()) {
- if (perStorage.remove(rdbi.getBlock()) != null) {
+ ReceivedDeletedBlockInfo oldRdbi = perStorage.get(rdbi.getBlock());
+ if (oldRdbi != null && oldRdbi.getBlock().getGenerationStamp() <
rdbi.getBlock().getGenerationStamp()
Review Comment:
This fix still leave one unexpected case, consider the new entry's
generation stamp is less than the old one, it will put again at line 265 and
overwrite it, right?
How about the following patch? ADD it will be better to add new unit test to
verify this bugfix. Thanks.
```
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/IncrementalBlockReportManager.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/IncrementalBlockReportManager.java
@@ -57,6 +57,14 @@
this.dnMetrics = dnMetrics;
}
+ /**
+ * Get block info from this IBR.
+ * @return block info if it exists; otherwise, return null.
+ */
+ ReceivedDeletedBlockInfo get(Block block) {
+ return blocks.getOrDefault(block, null);
+ }
+
/**
* Remove the given block from this IBR
* @return true if the block was removed; otherwise, return false.
@@ -252,8 +260,19 @@ synchronized void addRDBI(ReceivedDeletedBlockInfo rdbi,
// Make sure another entry for the same block is first removed.
// There may only be one such entry.
for (PerStorageIBR perStorage : pendingIBRs.values()) {
- if (perStorage.remove(rdbi.getBlock()) != null) {
- break;
+ ReceivedDeletedBlockInfo oldRdbi = perStorage.get(rdbi.getBlock());
+ if (oldRdbi != null) {
+ long oldGS = oldRdbi.getBlock().getGenerationStamp();
+ long newGS = rdbi.getBlock().getGenerationStamp();
+ // If the same block entry has existed and generation stamp less
than
+ // the new one, then remove it first. If generation stamp greater
than
+ // the new one, then keep it. Please reference HDFS-16922 for more
+ // details.
+ if (oldGS < newGS && perStorage.remove(rdbi.getBlock()) != null) {
+ break;
+ } else if (oldGS >= newGS) {
+ return;
+ }
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
