LogFormatter

[Daniel Feinstein]

Hi,   This is my first email to the community. In our (rawsugar.com) project we integrated multiple lucene indexes and a nutch package. We use the packages for about two years. Lucene is working great and we could use it as a "black box". Unfortunately, we had to modify Nutch package for our usage because of some leaks and a design problem in LogFormatter class. This fact force us to use a customized version of Nutch and avoid to update versions. At the last week we decided to move to the latest version of Nutch but discovered that the LogFormatter still have the same problem (leaks I believe has been fixed).   The problem: In Nutch project LogFormatter has duplicated functionality: 1) Logger records format and 2) Severe error handler   The first usage is standard and usually could be overwritten by a user of the package by modifying logging.properties file. The second usage is much more problematic because it affects the behavior of the whole application (not only Nutch package). To support the error handling LogFormatter enforce usage of the formatter class by all classes of the whole application which uses Nutch package. This is done by overwriting all the system handlers (class java.util.logging.Handler). This operation prevents the application to use its own log formatter. Also this cause LogFormatter.hasLoggedSevere() to be sensitive to all severe records in the big system but not only to relevant. More than that this flag, LogFormatter.loggedSevere is never cleaned what means if an application had one, even unrelated severe record, tools like Fetcher will never run until the application will be restarted.   I would like to suggest the following solutions: 1) To separate the functionality of log formatting and error handling or 2) Change LogFormatter class to be affected only by nutch package functions   For my opinion the first solution is much better especially if error handling will be encapsulated for each task. I have found the following usages of LogFormatter.hasLoggedSevere(): - Fetcher - URLFilterChecker - ParseSegment Unfortunately I'm not familiar enough with the usages above to implement this solution that why I suggest the second one. The attached file is our implementation of LogFormatter class which we use for more than a year. I hope this change will be accepted by the community.   Daniel _____________________ Daniel Feinstein Lead Software Engineer www.rawsugar.com   __________________________ CONFIDENTIALITY NOTICE This email may contain confidential and privileged material for the sole use of the intended recipient. Any review or distribution by others is strictly prohibited. If you are not the intended recipient please contact the sender and delete all copies.