ChengbingLiu opened a new pull request, #5246: URL: https://github.com/apache/hadoop/pull/5246
### Description of PR In our production cluster with Observer NameNode enabled, we have plenty of logs printed by `FSEditLogLoader` and `RedundantEditLogInputStream`. The `LogThrottlingHelper` doesn't seem to work. ``` 2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688] maxTxnsToRead = 92233720368547758072022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688]' to transaction ID 17686250688 2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688]' to transaction ID 17686250688 2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688]) of total size 527.0, total edits 1.0, total load time 0.0 ms 2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693] maxTxnsToRead = 9223372036854775807 2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693]' to transaction ID 17686250689 2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693]' to transaction ID 17686250689 2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693]) of total size 890.0, total edits 5.0, total load time 1.0 ms ``` After some digging, I found the cause is that `LogThrottlingHelper`'s are declared as instance variables of all the enclosing classes, including `FSImage`, `FSEditLogLoader` and `RedundantEditLogInputStream`. Therefore the logging frequency will not be limited across different instances. For classes with only limited number of instances, such as `FSImage`, this is fine. For others whose instances are created frequently, such as `FSEditLogLoader` and `RedundantEditLogInputStream`, it will result in plenty of logs. This can be fixed by declaring `LogThrottlingHelper`'s as static members. ### How was this patch tested? Through a test case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
