what do you consider 'huge' ?
Daniel Clark wrote:
I'm trying to index a huge URL list using nutch with hadoop on a two machine
cluster. It appears to be timing out at the injector. I get the following
error. I run a smaller URL list with no problems. Any help would be
appreciated.
Injector: starting
Injector: crawlDb: crawl7/crawldb
Injector: urlDir: urls/big_list.txt
Injector: Converting injected urls to crawl db entries.
task_0009_m_000001_0: log4j:ERROR setFile(null,true) call failed.
task_0009_m_000001_0: java.io.FileNotFoundException:
/home/d/daclark/nutch/search/nutch-0.9/logs (Is a directory)
task_0009_m_000001_0: at java.io.FileOutputStream.openAppend(Native
Method)
task_0009_m_000001_0: at
java.io.FileOutputStream.<init>(FileOutputStream.java:177)
task_0009_m_000001_0: at
java.io.FileOutputStream.<init>(FileOutputStream.java:102)
task_0009_m_000001_0: at
org.apache.log4j.FileAppender.setFile(FileAppender.java:289)
task_0009_m_000001_0: at
org.apache.log4j.FileAppender.activateOptions(FileAppender.java:163)
task_0009_m_000001_0: at
org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAp
pender.java:215)
task_0009_m_000001_0: at
org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:256)
task_0009_m_000001_0: at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:132
)
task_0009_m_000001_0: at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:96)
task_0009_m_000001_0: at
org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.jav
a:654)
task_0009_m_000001_0: at
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.jav
a:612)
~~~~~~~~~~~~~~~~~~~~~
Daniel Clark, President
DAC Systems, Inc.
(703) 403-0340
~~~~~~~~~~~~~~~~~~~~~
