Since hadoop's map files are write once, it is not possible to delete
some urls from the crawldb and linkdb. The only thing you can do is to
create the map files once again without the deleted urls. But running
the crawl once more as you suggested seems more appropriate. Deleting
documents from the index is just lucene stuff.
In your case it seems that every once in a while, you crawl the whole
site, and create the indexes and db's and then just throw the old one
out. And between two crawls you can delete the urls from the index.
[EMAIL PROTECTED] wrote:
Hi,
I'd like to be able to immediately remove certain pages from Nutch (index,
crawldb, linkdb...).
The scenario is that I'm using Nutch to index a single site or a set of
internal sites. Once in a while editors of the site remove a page from the
site. When that happens, I want to update at least the index and ideally
crawldb, linkdb, so that people searching the index don't get the missing page
in results and end up going there, hitting the 404.
I don't think there is a "direct" way to do this with Nutch, is there?
If there really is no direct way to do this, I was thinking I'd just put the
URL of the recently removed page into the first next fetchlist and then somehow
get Nutch to immediately remove that page/URL once it hits a 404. How does
that sound?
Is there a way to configure Nutch to delete the page after it gets a 404 for it
even just once? I thought I saw the setting for that somewhere a few weeks
ago, but now I can't find it.
Thanks,
Otis
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/ - Tag - Search - Share
