Rod Taylor wrote:
On Thu, 2006-03-09 at 12:09 -0800, Doug Cutting wrote:
Rod Taylor wrote:
First is to allow for cleaning up. This consists of a new option to
"updatedb" which can scrub the database of all URLs which no longer
match URLFilter settings (regex-urlfilter.txt). This allows a change in
the urlfilter to be reflected against Nutches current dataset, something
I think others have asked for in the past.
Yes, this would be a welcome addition. Note that Andrzej recently
committed a change that causes Generate to filter urls, which achieves
the same effect, but w/o removing them from the database, so they're
still consuming space & time.
Excellent. I'll put someone on this.
Stefan submitted some code that from my cursory glance looks good -
could you please check NUTCH-226 and see if it works for you? I plan to
run some tests and add this (plus a companion LinkDBFilter).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
