Aaron Guo created HDFS-15292:
--------------------------------
Summary: Infinite loop in Lease Manager due to replica is missing
in dn
Key: HDFS-15292
URL: https://issues.apache.org/jira/browse/HDFS-15292
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.1.3
Reporter: Aaron Guo
In our production environment, we found that files of under construction keep
growing, and the lease manager is trying to release the lease in a Infinite
loop:
{code:java}
2020-04-18 23:10:57,816 WARN namenode.LeaseManager
(LeaseManager.java:checkLeases(589)) - Cannot release the path
/user/hadoop/myTestFile.txt in the lease [Lease. Holder:
go-hdfs-7VVGF3sGvHZcsZZC, pending creates: 1]. It will be retried.
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: DIR*
NameSystem.internalReleaseLease: Failed to release lease for file
/user/hadoop/myTestFile.txt. Committed blocks are waiting to be minimally
replicated. Try again later.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.internalReleaseLease(FSNamesystem.java:3391)
at
org.apache.hadoop.hdfs.server.namenode.LeaseManager.checkLeases(LeaseManager.java:586)
at
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:524)
at java.lang.Thread.run(Thread.java:745)
{code}
This is because the last block of this file can NOT meet the minimum required
replica of 1, a AlreadyBeingCreatedException get thrown, and it will keeps
retry forever.
This infinite loop also cause another issue since the lease manager always
trying to release the first lease then goto the next one, so no lease will be
released.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
