[
https://issues.apache.org/jira/browse/HDFS-16408?focusedWorklogId=703309&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703309
]
ASF GitHub Bot logged work on HDFS-16408:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Jan/22 13:59
Start Date: 04/Jan/22 13:59
Worklog Time Spent: 10m
Work Description: liever18 commented on a change in pull request #3853:
URL: https://github.com/apache/hadoop/pull/3853#discussion_r778100897
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
##########
@@ -530,35 +530,37 @@ public void setLeasePeriod(long softLimit, long
hardLimit) {
/** Check leases periodically. */
@Override
public void run() {
- for(; shouldRunMonitor && fsnamesystem.isRunning(); ) {
- boolean needSync = false;
- try {
- // sleep now to avoid infinite loop if an exception was thrown.
- Thread.sleep(fsnamesystem.getLeaseRecheckIntervalMs());
-
- // pre-filter the leases w/o the fsn lock.
- Collection<Lease> candidates = getExpiredCandidateLeases();
- if (candidates.isEmpty()) {
- continue;
- }
-
- fsnamesystem.writeLockInterruptibly();
+ try {
+ for (; shouldRunMonitor && fsnamesystem.isRunning(); ) {
+ boolean needSync = false;
try {
- if (!fsnamesystem.isInSafeMode()) {
- needSync = checkLeases(candidates);
+ // sleep now to avoid infinite loop if an exception was thrown.
+ Thread.sleep(fsnamesystem.getLeaseRecheckIntervalMs());
+
+ // pre-filter the leases w/o the fsn lock.
+ Collection<Lease> candidates = getExpiredCandidateLeases();
+ if (candidates.isEmpty()) {
+ continue;
}
- } finally {
- fsnamesystem.writeUnlock("leaseManager");
- // lease reassignments should to be sync'ed.
- if (needSync) {
- fsnamesystem.getEditLog().logSync();
+
+ fsnamesystem.writeLockInterruptibly();
+ try {
+ if (!fsnamesystem.isInSafeMode()) {
+ needSync = checkLeases(candidates);
+ }
+ } finally {
+ fsnamesystem.writeUnlock("leaseManager");
+ // lease reassignments should to be sync'ed.
+ if (needSync) {
+ fsnamesystem.getEditLog().logSync();
+ }
}
+ } catch (InterruptedException ie) {
+ LOG.debug("{} is interrupted", name, ie);
}
- } catch(InterruptedException ie) {
- LOG.debug("{} is interrupted", name, ie);
- } catch(Throwable e) {
- LOG.warn("Unexpected throwable: ", e);
}
+ } catch (Throwable e) {
Review comment:
> I think we should revert this part of the change, but keep the new
range check you added instead.
I totally agree with your point of view
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 703309)
Remaining Estimate: 0h (was: 10m)
Time Spent: 1h (was: 50m)
> Negative LeaseRecheckIntervalMs will let LeaseMonitor loop forever and print
> huge amount of log
> -----------------------------------------------------------------------------------------------
>
> Key: HDFS-16408
> URL: https://issues.apache.org/jira/browse/HDFS-16408
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.1.3, 3.3.1
> Reporter: Jingxuan Fu
> Priority: Major
> Labels: pull-request-available
> Original Estimate: 1h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> There is a problem with the try catch statement in the LeaseMonitor daemon
> (in LeaseManager.java), when an unknown exception is caught, it simply prints
> a warning message and continues with the next loop.
> An extreme case is when the configuration item
> 'dfs.namenode.lease-recheck-interval-ms' is accidentally set to a negative
> number by the user, as the configuration item is read without checking its
> range, 'fsnamesystem. getLeaseRecheckIntervalMs()' returns this value and is
> used as an argument to Thread.sleep(). A negative argument will cause
> Thread.sleep() to throw an IllegalArgumentException, which will be caught by
> 'catch(Throwable e)' and a warning message will be printed.
> This behavior is repeated for each subsequent loop. This means that a huge
> amount of repetitive messages will be printed to the log file in a short
> period of time, quickly consuming disk space and affecting the operation of
> the system.
> As you can see, 178M log files are generated in one minute.
>
> {code:java}
> ll logs/
> total 174456
> drwxrwxr-x 2 hadoop hadoop 4096 1月 3 15:13 ./
> drwxr-xr-x 11 hadoop hadoop 4096 1月 3 15:13 ../
> -rw-rw-r-- 1 hadoop hadoop 36342 1月 3 15:14
> hadoop-hadoop-datanode-ljq1.log
> -rw-rw-r-- 1 hadoop hadoop 1243 1月 3 15:13
> hadoop-hadoop-datanode-ljq1.out
> -rw-rw-r-- 1 hadoop hadoop 178545466 1月 3 15:14
> hadoop-hadoop-namenode-ljq1.log
> -rw-rw-r-- 1 hadoop hadoop 692 1月 3 15:13
> hadoop-hadoop-namenode-ljq1.out
> -rw-rw-r-- 1 hadoop hadoop 33201 1月 3 15:14
> hadoop-hadoop-secondarynamenode-ljq1.log
> -rw-rw-r-- 1 hadoop hadoop 3764 1月 3 15:14
> hadoop-hadoop-secondarynamenode-ljq1.out
> -rw-rw-r-- 1 hadoop hadoop 0 1月 3 15:13 SecurityAuth-hadoop.audit
>
> tail -n 15 logs/hadoop-hadoop-namenode-ljq1.log
> 2022-01-03 15:14:46,032 WARN
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Unexpected throwable:
> java.lang.IllegalArgumentException: timeout value is negative
> at java.base/java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:534)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-01-03 15:14:46,033 WARN
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Unexpected throwable:
> java.lang.IllegalArgumentException: timeout value is negative
> at java.base/java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:534)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-01-03 15:14:46,033 WARN
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Unexpected throwable:
> java.lang.IllegalArgumentException: timeout value is negative
> at java.base/java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:534)
> at java.base/java.lang.Thread.run(Thread.java:829)
> {code}
>
> I think there are two potential solutions.
> The first is to adjust the position of the try catch statement in the
> LeaseMonitor daemon by moving 'catch(Throwable e)' to the outside of the loop
> body. This can be done like the NameNodeResourceMonitor daemon, which ends
> the thread when an unexpected exception is caught.
> The second is to use Precondition.checkArgument() to scope the configuration
> item 'dfs.namenode.lease-recheck-interval-ms' when it is read, to avoid the
> wrong configuration item can affect the subsequent operation of the program.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
