Hey Michael, please see inline..
On Sat, 27 Aug 2005 14:54:21 -0700 (PDT), Michael Ji wrote:
> hi Kelvin:
>
> Thanks your hint for depth control. I will try it tonight and will
> let you know the result.
>
> I guess the design of patch-84 is to become an independent crawler
> by itself. Is it true?
>
nutch-84 was designed to be a standalone focused/constrained crawler.
> So, it will replace the commands of "nutch/admintool create..,
> nutch/generate, nutch/updateda", etc, by only using OC APIs.
>
No. Nutch is more than just the crawler (webdb, analyzer, etc). nutch-84 was
created for people who want a crawler (to use with lucene) but not necessarily
the whole nutch infrastructure. Take for instance the fact that the nutch query
language is slighty different from Lucene's. By extending this simple
PostFetchProcessor below, you can easily just crawl and add documents directly
to a lucene index without needing to use WebDB (or the bin/nutch/index command).
public abstract class LucenePostFetchProcessor implements PostFetchProcessor{
private String index;
private IndexWriter writer;
private boolean overwrite;
private Analyzer analyzer;
public void process(FetcherOutput fo, Content content, Parse parse)
throws IOException {
if(writer == null) initWriter();
writeDocument(fo, content, parse);
}
protected abstract void writeDocument(
FetcherOutput fo, Content content, Parse parse);
private void initWriter() throws IOException {
writer = new IndexWriter(index, analyzer, overwrite);
}
public void close() throws IOException {
if(writer != null) writer.close();
}
// setters
}
> I mean, OC can form its own fetch list for the fetching next round,
> for example. Only the fetched result needs to be indexed and merged.
>
If what you mean is that OC will run continuously until there are no more URLs
to fetch, you are correct. Unfortunately, until we deal with the problem of bot
traps, I don't think this is a good idea for a production environment.
HTH,
k
-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
