Re: Crawl dies unexpectedly

matt davies Tue, 01 Apr 2008 00:34:57 -0700

Hi Susam

I think that is exactly what happened, I stopped a crawl and there wasthis bogus segment folder.


I then deleted the folder, ran it again and it all worked fine.

With recrawling what is the benefit?

Wouldn't it be a better/cleaner solution to delete the crawl folderevery night and crawlf from scratch?

That way I'm not going to have to clean up segments at some point inthe future.


The crawl takes 40 minutes in total so it's not a biggie.

What do you think Susam?

matt


On 31 Mar 2008, at 18:13, Susam Pal wrote:

Hi,

You seem to be using the latest revision from trunk. In the commit for
revision #637122, recrawling was introduced. So, you can crawl using
the same 'crawl' directory more than once. If you do a crawl with
-depth M in the first crawl and -depth N again, you'll end up with M +
N segments.

My guess is that you might have stopped the first crawl before
completion and the segment which remained incomplete caused the error.
If my guess is right, you would probably get the same error again due
to the same segment. If it happens, you might have to delete that
segment to proceed with the index generation.

Regards,
Susam Pal

On Mon, Mar 31, 2008 at 7:14 PM, matt davies <[EMAIL PROTECTED]>wrote:

Hi Dennis

"If you have a crawl depth of 3 then there should be only 3segments/*

folder"

Thanks for that titbit, that makes a bit more sense now.

I have no idea where the other ones are coming from.

One of the sites I'm scanning is quit large, more than 10,000 pages,
in total we're talking about roughly 20,000 pages.

What would you recommend setting the crawl depth to Dennis?

I've tried rerunning the crawl after deleting the entire folder that
it was jamming on, it seems to be crawling again.

See what happens this time

Thanks for getting back to me Dennis.





On 31 Mar 2008, at 14:37, Dennis Kubes wrote:

If you have a crawl depth of 3 then there should be only 3 segments/
* folders.  Any idea where the others came from?

Dennis

matt davies wrote:

Hello everyone
I've just added 12 urls to my urls/filename file and added the same
URLS to my craw-urlfilter.txt file and ran the crawl like so
bin/nutch crawl urls -dir crawl -depth 3
the crawl runs fine, it starts grabbing the urls and creating the
segments, but then all of a sudden it dies with the following error
when trying to merge the segments.
CrawlDb update: segments: [crawl/segments/20080331113907]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: done
LinkDb: starting
LinkDb: linkdb: crawl/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true