With recrawling, the benefit is that your index can go live sooner. That is, you can do a crawl and get an index that goes live. Then you can keep recrawling.
Deleting the crawl folder overnight may be feasible for you since your crawl is very short. For many of us, the crawl takes a few days to a week, so deleting the crawl is not always feasible for us. I crawl and re-crawl for a week and then start over with a new crawl directory every week. Is there a better way to manage the crawls? If someone has a better way, please share. Regards, Susam Pal On Tue, Apr 1, 2008 at 1:04 PM, matt davies <[EMAIL PROTECTED]> wrote: > Hi Susam > > I think that is exactly what happened, I stopped a crawl and there was > this bogus segment folder. > > I then deleted the folder, ran it again and it all worked fine. > > With recrawling what is the benefit? > > Wouldn't it be a better/cleaner solution to delete the crawl folder > every night and crawlf from scratch? > > That way I'm not going to have to clean up segments at some point in > the future. > > The crawl takes 40 minutes in total so it's not a biggie. > > What do you think Susam? > > matt > > > > > On 31 Mar 2008, at 18:13, Susam Pal wrote: > > > Hi, > > > > You seem to be using the latest revision from trunk. In the commit for > > revision #637122, recrawling was introduced. So, you can crawl using > > the same 'crawl' directory more than once. If you do a crawl with > > -depth M in the first crawl and -depth N again, you'll end up with M + > > N segments. > > > > My guess is that you might have stopped the first crawl before > > completion and the segment which remained incomplete caused the error. > > If my guess is right, you would probably get the same error again due > > to the same segment. If it happens, you might have to delete that > > segment to proceed with the index generation. > > > > Regards, > > Susam Pal > > > > On Mon, Mar 31, 2008 at 7:14 PM, matt davies <[EMAIL PROTECTED]> > > wrote: > >> Hi Dennis > >> > >> > >> "If you have a crawl depth of 3 then there should be only 3 > >> segments/* > >> folder" > >> > >> Thanks for that titbit, that makes a bit more sense now. > >> > >> I have no idea where the other ones are coming from. > >> > >> One of the sites I'm scanning is quit large, more than 10,000 pages, > >> in total we're talking about roughly 20,000 pages. > >> > >> What would you recommend setting the crawl depth to Dennis? > >> > >> I've tried rerunning the crawl after deleting the entire folder that > >> it was jamming on, it seems to be crawling again. > >> > >> See what happens this time > >> > >> Thanks for getting back to me Dennis. > >> > >> > >> > >> > >> > >> On 31 Mar 2008, at 14:37, Dennis Kubes wrote: > >> > >>> If you have a crawl depth of 3 then there should be only 3 segments/ > >>> * folders. Any idea where the others came from? > >>> > >>> Dennis > >>> > >>> matt davies wrote: > >>>> Hello everyone > >>>> I've just added 12 urls to my urls/filename file and added the same > >>>> URLS to my craw-urlfilter.txt file and ran the crawl like so > >>>> bin/nutch crawl urls -dir crawl -depth 3 > >>>> the crawl runs fine, it starts grabbing the urls and creating the > >>>> segments, but then all of a sudden it dies with the following error > >>>> when trying to merge the segments. > >>>> CrawlDb update: segments: [crawl/segments/20080331113907] > >>>> CrawlDb update: additions allowed: true > >>>> CrawlDb update: URL normalizing: true > >>>> CrawlDb update: URL filtering: true > >>>> CrawlDb update: Merging segment data into db. > >>>> CrawlDb update: done > >>>> LinkDb: starting > >>>> LinkDb: linkdb: crawl/linkdb > >>>> LinkDb: URL normalize: true > >>>> LinkDb: URL filter: true > >>>> LinkDb: adding segment: file:/home/nutch/nutch/trunk/crawl/ > >>>> segments/ > >>>> 20080331112151 > >>>> LinkDb: adding segment: file:/home/nutch/nutch/trunk/crawl/ > >>>> segments/ > >>>> 20080331111831 > >>>> LinkDb: adding segment: file:/home/nutch/nutch/trunk/crawl/ > >>>> segments/ > >>>> 20080331111720 > >>>> LinkDb: adding segment: file:/home/nutch/nutch/trunk/crawl/ > >>>> segments/ > >>>> 20080331111741 > >>>> LinkDb: adding segment: file:/home/nutch/nutch/trunk/crawl/ > >>>> segments/ > >>>> 20080331113907 > >>>> Exception in thread "main" > >>>> org.apache.hadoop.mapred.InvalidInputException: Input path doesnt > >>>> exist : file:/home/nutch/nutch/trunk/crawl/segments/20080331111741/ > >>>> parse_data > >>>> at > >>>> org > >>>> .apache > >>>> .hadoop.mapred.FileInputFormat.validateInput(FileInputFormat.java: > >>>> 154) at > >>>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:537) > >>>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:805) > >>>> at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:170) > >>>> at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:147) > >>>> at org.apache.nutch.crawl.Crawl.main(Crawl.java:129) > >>>> I checked one of the other segments, 20080331111720, and this > >>>> contained the following data > >>>> drwxr-xr-x 3 nutch nutch 4096 2008-03-31 11:17 content > >>>> drwxr-xr-x 3 nutch nutch 4096 2008-03-31 11:17 crawl_fetch > >>>> drwxr-xr-x 2 nutch nutch 4096 2008-03-31 11:17 crawl_generate > >>>> drwxr-xr-x 2 nutch nutch 4096 2008-03-31 11:17 crawl_parse > >>>> drwxr-xr-x 3 nutch nutch 4096 2008-03-31 11:17 parse_data > >>>> drwxr-xr-x 3 nutch nutch 4096 2008-03-31 11:17 parse_text > >>>> But the segment with the problem in does not contain all that data, > >>>> only > >>>> drwxr-xr-x 2 nutch nutch 4096 2008-03-31 11:17 crawl_generate > >>>> has anyone got any ideas what could be going wrong here? I've > >>>> checked space issues, loads of gigs free, and permissions on the > >>>> folders are identical. > >>>> Here's my nutch svn details > >>>> [EMAIL PROTECTED]:~/nutch/trunk$ svn info > >>>> Path: . > >>>> URL: http://svn.apache.org/repos/asf/lucene/nutch/trunk > >>>> Repository Root: http://svn.apache.org/repos/asf > >>>> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68 > >>>> Revision: 641752 > >>>> Node Kind: directory > >>>> Schedule: normal > >>>> Last Changed Author: ab > >>>> Last Changed Rev: 638782 > >>>> Last Changed Date: 2008-03-19 10:45:55 +0000 (Wed, 19 Mar 2008) > >>>> Any help, greatly appreciated. > >> > >> > >
