HI,
I think that the best way add step and chekpoint in the reduce part.
The first step crawl, make a ligth chekpoint.
The second map.
The three reduce, make a chekpoint.
If you have a error, return to checkpoint.
like a batch job.
Thanks JuanP
Andrzej Bialecki
<[EMAIL PROTECTED]>
Para
22/05/2006 16:34 [email protected]
cc
Por favor, Asunto
responda a Re: Restarting Just Reduce Part of
[EMAIL PROTECTED] Fetch
.apache.org
Dennis Kubes wrote:
> Is there a way to restart just the reduce part of the fetching process
> if it failed. The Map jobs for 5M pages completed but the reduce jobs
> failed late. Is there a way to recover this so I don't have to
> recrawl all 5M pages?
I'm sorry to say but it's not possible at the moment (I know the feeling
... :-/ ). This is a limitation in Hadoop, there have been recently some
(as yet inconclusive) discussions on this.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
