On 2010-03-13 00:12, Abhi Yerra wrote:
So I had -noParsing set. So parsing was not part of the fetch. The pages have been crawled, but the reducers have crashed. So if I restart the fetch will it try to crawl all those pages again?
Yes. It would be good to investigate first Why it crashed, otherwise it's likely to happen again. Are you running this on a cluster? Check the logs of the crashed tasks (in logs/userlogs/ on respective tasktracker nodes).
-- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com
