> Following the tutorial, I redirect the log messages to a log file. But, > when crawling 1 million pages, this log file can become hugh and writing > log messages to a huge file can slow down the fetching process. Is > there a better way to manage the log? maybe saving it to a series of > smaller files? appreciate your suggestions.
Change your JDK logging behaviors in $JAVA_HOME/jre/lib/logging.properties and add a FileHandler with limit greater than 0. Thus, the logs will be rotated in many log files (with a maximum size = limit). See also http://java.sun.com/j2se/1.4.2/docs/api/java/util/logging/package-summary.html http://java.sun.com/j2se/1.4.2/docs/api/java/util/logging/FileHandler.html Regards Jérôme -- http://motrech.free.fr/ http://www.frutch.org/
