Hi there,
Actually, it is my coding issue.
In parseOutputSegment.java, I manipulate the target crawlDatum after Page
Ranking function call. Somehow, this inproper order causes the indexer
failure.
Sorry to bother,
Michael,
On 8/11/06, Feng Ji <[EMAIL PROTECTED]> wrote:
Hi there,
Based on nutch-0.8. release package, I did some customerization on
ParseData's Content. Essentially, I adding data to Content's (Metadata)
metadata in fetcher and retrieve the information in ParseOutputFormat.
I did crawling testing, injector, generator, fetcher all works fine,
except index gives the following errors.
"
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:357)
at org.apache.nutch.indexer.Indexer.index(Indexer.java:296)
at org.apache.nutch.indexer.Indexer.main(Indexer.java:313)
"
1.
It seems a DFS I/O failure. This failure is not happened for all the
cases. For crawling some sites, index works and for other sites, it failed.
(I tested on hadoop 4.0.jar and hadoop5.0.jar, and get the same result)
2.
I crawl by nutch-0.8. package without modification, then everything works
perfectly. I wonder the reason is because I add data in Content's metadata
in parsed segment, and that causes index failure sometime. (not all the
time)
Any one has any idea?
thanks your time,
Michael,
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
