I have a 2 node testing cluster, I can no longer run the generate
task with out a log filled with errors (after about 19 successful
iterations)
running
./bin/nutch generate crawl/crawldb testsegments -topN 10
generates the following
----
2007-02-01 15:28:37,510 WARN fs.FSNamesystem - Replication requested
of 10 is larger than cluster size (2). Using cluster size.
2007-02-01 15:28:37,510 DEBUG dfs.StateChange - DIR*
NameSystem.startFile: add /home/nutch/nutch/filesystem/mapred/system/
submit_l03eux/.job.xml.crc to pendingCreates for DFSClient_-980095089
2007-02-01 15:28:37,511 DEBUG dfs.StateChange - BLOCK*
NameSystem.allocateBlock: /home/nutch/nutch/filesystem/mapred/system/
submit_l03eux/.job.xml.crc. blk_4008532917963069756 is created and
added to pendingCreates and pendingCreateBlocks
2007-02-01 15:28:37,594 DEBUG dfs.DataNode - Number of active
connections is: 1
2007-02-01 15:28:37,627 INFO dfs.DataNode - Received block
blk_4008532917963069756 from /192.168.10.95
----
After this the log is filled with endless groups of 8 fs.FSNamesystem
- Could not find any nodes with sufficient capacity
(10 replicas requested - 2 slaves?)
This seems to put the server in a loop of eating up all the CPU as
the tasks run leaving very little for the actual jobs
----
2007-02-01 15:28:40,786 INFO mapred.JobClient - Running job: job_0001
2007-02-01 15:28:41,328 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:41,328 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:41,328 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:41,328 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:41,678 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:41,678 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:41,678 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:41,678 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:41,794 INFO mapred.JobClient - map 0% reduce 0%
2007-02-01 15:28:44,320 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:44,320 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:44,320 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:44,320 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:44,699 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:44,699 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:44,699 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
2007-02-01 15:28:44,699 WARN fs.FSNamesystem - Could not find any
nodes with sufficient capacity
Can anyone provide any help with this problem?
These continue even after I kill the generate task
Jason
