Hello!
I am brand new to Nutch, so please excuse my ignorance on these things.
Basically, if I would want to provide an "Add URL" link for my customers,
how would I go about implementing this? I should be adding the URL in a file
in the urls/ directory, and possibly the crawl-urlfilter.txt file as well, I
would
assume.
And then I should re-index somehow, or - ideally - just do a crawl of
that new site, with the results then somehow merged into the old craw
results.
But that seems to be astonishingly difficult to get working.
After googling around about 're-crawling' or 'index updating' for a while,
it appears to me as if there is no one way to do this, which just works
for everyone. There are some scripts posted, but they only seem to work
for some people. Same with me. The basic re-crawler/index scripts I could
find don't work for me for some reason or the other.
I tried to use the command tools for merging crawl results, something
like this:
$ bin/nutch org.apache.nutch.crawl.CrawlDbMerger ...
$ bin/nutch org.apache.nutch.segment.SegmentMerger ...
$ bin/nutch org.apache.nutch.crawl.LinkDbMerger ...
And that seemed to work ok, but then I had to do the indexes as the
last step:
$ bin/nutch org.apache.nutch.indexer.IndexMerger ...
And there maybe I am doing something wrong, but I find that in my old
and new crawl directory I have an index/ and indexes/ directory and now
I don't know how to use the IndexMerger, since it doesn't seem to want
to create both of those directories.
Are those four command really the only way to go about it? Is there
something easier or better?
For what it's worth: I'm using Nutch 0.9.
Thank you very much!
--
View this message in context:
http://www.nabble.com/How-implement-an-%22add-URL%22-with-Nutch--Or%3A-Updating-the-index-crawl-db-tp17323793p17323793.html
Sent from the Nutch - User mailing list archive at Nabble.com.
