Aaron Guo created HDFS-15292: -------------------------------- Summary: Infinite loop in Lease Manager due to replica is missing in dn Key: HDFS-15292 URL: https://issues.apache.org/jira/browse/HDFS-15292 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.1.3 Reporter: Aaron Guo
In our production environment, we found that files of under construction keep growing, and the lease manager is trying to release the lease in a Infinite loop: {code:java} 2020-04-18 23:10:57,816 WARN namenode.LeaseManager (LeaseManager.java:checkLeases(589)) - Cannot release the path /user/hadoop/myTestFile.txt in the lease [Lease. Holder: go-hdfs-7VVGF3sGvHZcsZZC, pending creates: 1]. It will be retried. org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: DIR* NameSystem.internalReleaseLease: Failed to release lease for file /user/hadoop/myTestFile.txt. Committed blocks are waiting to be minimally replicated. Try again later. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.internalReleaseLease(FSNamesystem.java:3391) at org.apache.hadoop.hdfs.server.namenode.LeaseManager.checkLeases(LeaseManager.java:586) at org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:524) at java.lang.Thread.run(Thread.java:745) {code} This is because the last block of this file can NOT meet the minimum required replica of 1, a AlreadyBeingCreatedException get thrown, and it will keeps retry forever. This infinite loop also cause another issue since the lease manager always trying to release the first lease then goto the next one, so no lease will be released. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org