[
https://issues.apache.org/jira/browse/NUTCH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573790#action_12573790
]
Susam Pal commented on NUTCH-601:
---------------------------------
It continues the recrawl using the existing crawl directory. It generates new
segments using the already existing crawl/crawldb directory. You can assume a
recrawl to be a crawl resumed after a break (taken for generating and merging
indexes). In other words, if you did two crawls with "depth" set as 5,
effectively you have done a crawl of depth 10.
I am not clear about what you mean by filtering. Isn't conf/crawl-urlfilter.txt
enough for what you want to filter in the second crawl?
> Recrawling on existing crawl directory using force option
> ---------------------------------------------------------
>
> Key: NUTCH-601
> URL: https://issues.apache.org/jira/browse/NUTCH-601
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Susam Pal
> Priority: Minor
> Attachments: NUTCH-601v0.1.patch, NUTCH-601v0.2.patch,
> NUTCH-601v0.3.patch, NUTCH-601v1.0.patch
>
>
> Added a '-force' option to the 'bin/nutch crawl' command line. With this
> option, one can crawl and recrawl in the following manner:
> {code}
> bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5
> bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force
> {code}
> This option can be used for the first crawl too:
> {code}
> bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force
> bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force
> {code}
> If one tries to crawl without the -force option when the crawl directory
> already exists, he/she finds a small warning along with the error message:
> {code}
> # bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5
> Exception in thread "main" java.lang.RuntimeException: crawl already
> exists. Add -force option to recrawl.
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:89)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
