Andrzej,
I updated nutch to the trunk last night and I split up a crawl of 1.6M
into 4 chunks of 400K using the updated generator. However, the first
crawl of 400K crashed last night with some new errors I have never
seen before:
org.apache.hadoop.fs.ChecksumException: Checksum Error
java.io.IOException: Could not obtain block:
blk_-8206810763586975866_5190 file=/user/hadoop/crawl/segments/
20091020170107/crawl_generate/part-00009
Do you know why I would be getting these errors? I had a lost tracker
error also - could these problems be related?
Thanks,
Eric
On Oct 20, 2009, at 2:13 PM, Andrzej Bialecki wrote:
Eric Osgood wrote:
This is the error I keep getting whenever I try to fetch more than
400K files at a time using a 4 node hadoop cluster running nutch 1.0.
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
failed to create file /user/hadoop/crawl/segments/20091013161641/
crawl_fetch/part-00015/index for
DFSClient_attempt_200910131302_0011_r_000015_2 on client
192.168.1.201 because current leaseholder is trying to recreate file.
Please see this issue:
https://issues.apache.org/jira/browse/NUTCH-692
Apply the patch that is attached there, rebuild Nutch, and tell me
if this fixes your problem.
(the patch will be applied to trunk anyway, since others confirmed
that it fixes this issue).
Can anybody shed some light on this issue? I was under the
impression that 400K was small potatoes for a nutch hadoop combo?
It is. This problem is rare - I think I crawled cumulatively ~500mln
pages in various configs and it didn't occur to me personally. It
requires a few things to go wrong (see the issue comments).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Eric Osgood
---------------------------------------------
Cal Poly - Computer Engineering, Moon Valley Software
---------------------------------------------
eosg...@calpoly.edu, e...@lakemeadonline.com
---------------------------------------------
www.calpoly.edu/~eosgood, www.lakemeadonline.com
