[EMAIL PROTECTED] wrote:
Don't generate URLs that don't pass URLFilters.
Just to be clear, this is to support folks changing their filters while they're crawling, right? We already filter before we put things into the db, so we're filtering twice now, no? If so, then perhaps there should be an option to disable this second filtering for folks who don't change their filters?
Doug
