I tried asking this over at the nutch-user alias, but I am seeing very
little traction, so I thought I'd ask the developers. I realize this is most
likely a configuration problem on my end, but I am very new to using nutch,
so I am having a difficult time understanding where I need to look.
Does anyone have any insight into the following error I am seeing in the
hadoop logs? Is this something I should be concerned with, or is it expected
that this shows up in the logs from time to time? If it is not expected,
where can I look for more information on what is going on?
2009-10-16 17:02:43,061 ERROR datanode.DataNode -
DatanodeRegistration(192.168.1.7:50010,
storageID=DS-1226842861-192.168.1.7-50010-1254609174303,
infoPort=50075, ipcPort=50020):DataXceiver
org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException:
Block blk_909837363833332565_3277 is valid, and cannot be written to.
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:975)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:97)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Thread.java:636)
I am able to produce this just injecting the urls (2 of them), but it shows
up on both datanodes, and happens whenever I run an operation that uses dfs.
I am running the latest sources from the trunk.
I've verified that only one instance of the following on the datanodes:
org.apache.hadoop.hdfs.server.datanode.DataNode
org.apache.hadoop.mapred.TaskTracker
I've also verified that only one instance of the following are running on
the name node:
org.apache.hadoop.hdfs.server.namenode.NameNode
org.apache.hadoop.mapred.JobTracker
The hardware is as follows:
Two data nodes, both configured identical. Atom 330 proc, 2gigs ram, 320g
SATA 3.0 hard drive, Fedora Core 10.
One name node, running some amd x86 proc, 2 gigs memory, 750g SATA, Fedora
Core 10. (pieced together from spare parts)
All across a 100mb network.
Admittedly this is low end hardware, but I am doing this specifically as an
exercise in using low power (as in electricity) hardware.
I can also provide config files if needed.
Jesse
int GetRandomNumber()
{
return 4; // Chosen by fair roll of dice
// Guaranteed to be random
} // xkcd.com
