Thanks Mathijs, it was exactly what i needed. We should create a JIRA to add on the next Nutch release.
Hi Emmanuel, I think it has something to do with the fact that the environment variables ${hadoop.log.dir} and ${hadoop.log.file} are not propagated to the child JVM's which are spawned by the TaskTracker. The log4j.properties file which comes with *Hadoop* solves this by putting the following lines at the top of the log4j.properties file: # Define some default values that can be overridden by system properties hadoop.log.dir=. hadoop.log.file=hadoop.log I don't know why the Nutch version of the log4j.properties file hasn't got this in it. Anyway, adding this solved it for me. Mathijs Emmanuel JOKE wrote:Hi Guys, I have a cluster of 2 machine : Linux; Java 1.6 I started a crawl on a list of few website only. I used the command bin/nutch crawl urls/site1 -dir crawld -depth 10 -topN 100000 -threads 30 I had an error on my 6th depth. CrawlDb update: starting CrawlDb update: db: crawld/crawldb CrawlDb update: segments: [crawld/segments/20070627053531] CrawlDb update: additions allowed: true CrawlDb update: URL normalizing: true CrawlDb update: URL filtering: true CrawlDb update: Merging segment data into db. task_0035_m_000005_0: log4j:ERROR setFile(null,true) call failed. task_0035_m_000005_0: java.io.FileNotFoundException: /data/sengine/search/bin/../logs (Is a directory) task_0035_m_000005_0: at java.io.FileOutputStream.openAppend(Native Method) task_0035_m_000005_0: at java.io.FileOutputStream.<init>( FileOutputStream.java:177) task_0035_m_000005_0: at java.io.FileOutputStream.<init>( FileOutputStream.java:102) task_0035_m_000005_0: at org.apache.log4j.FileAppender.setFile( FileAppender.java:289) task_0035_m_000005_0: at org.apache.log4j.FileAppender.activateOptions( FileAppender.java:163) task_0035_m_000005_0: at org.apache.log4j.DailyRollingFileAppender.activateOptions( DailyRollingFileAppender.java:215) task_0035_m_000005_0: at org.apache.log4j.config.PropertySetter.activate( PropertySetter.java:256) task_0035_m_000005_0: at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java :132) task_0035_m_000005_0: at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java
:96)
task_0035_m_000005_0: at org.apache.log4j.PropertyConfigurator.parseAppender( PropertyConfigurator.java:654) task_0035_m_000005_0: at org.apache.log4j.PropertyConfigurator.parseCategory( PropertyConfigurator.java:612) task_0035_m_000005_0: at org.apache.log4j.PropertyConfigurator.configureRootCategory( PropertyConfigurator.java:509) task_0035_m_000005_0: at org.apache.log4j.PropertyConfigurator.doConfigure (PropertyConfigurator.java:415) task_0035_m_000005_0: at org.apache.log4j.PropertyConfigurator.doConfigure (PropertyConfigurator.java:441) task_0035_m_000005_0: at org.apache.log4j.helpers.OptionConverter.selectAndConfigure( OptionConverter.java:468) task_0035_m_000005_0: at org.apache.log4j.LogManager.<clinit>( LogManager.java:122) task_0035_m_000005_0: at org.apache.log4j.Logger.getLogger(Logger.java :104) task_0035_m_000005_0: at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java
:229)
task_0035_m_000005_0: at org.apache.commons.logging.impl.Log4JLogger .<init>(Log4JLogger.java:65) task_0035_m_000005_0: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) task_0035_m_000005_0: at sun.reflect.NativeConstructorAccessorImpl.newInstance( NativeConstructorAccessorImpl.java:39) task_0035_m_000005_0: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( DelegatingConstructorAccessorImpl.java:27) task_0035_m_000005_0: at java.lang.reflect.Constructor.newInstance( Constructor.java:513) task_0035_m_000005_0: at org.apache.commons.logging.impl.LogFactoryImpl.newInstance( LogFactoryImpl.java:529) task_0035_m_000005_0: at org.apache.commons.logging.impl.LogFactoryImpl.getInstance( LogFactoryImpl.java:235) task_0035_m_000005_0: at org.apache.commons.logging.LogFactory.getLog( LogFactory.java:370) task_0035_m_000005_0: at org.apache.hadoop.mapred.TaskTracker.<clinit>( TaskTracker.java:82) task_0035_m_000005_0: at org.apache.hadoop.mapred.TaskTracker$Child.main( TaskTracker.java:1423) task_0035_m_000005_0: log4j:ERROR Either File or DatePattern options are not set for appender [DRFA]. task_0035_m_000004_0: log4j:ERROR setFile(null,true) call failed. I notice that every time I had a nutch error when i use my cluster it give me this exception regardings the log4j. I don't understand why. Any idea ? Besides my crawl stopped and I don't why too. How can i get any details ? cheers E-- Knowlogy Helperpark 290 C 9723 ZA Groningen [EMAIL PROTECTED] +31 (0)6 15312977 http://www.knowlogy.nl
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
_______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
