Re: blacklist for crawling

2010-01-26 Thread James Todd
Have you looked into the regex-urlfilter file? You can specify positive and negative URL patterns to filter against. - james On Jan 26, 2010 5:01 PM, "Ted Yu" wrote: Hi, Is there feature that would enable blacklisting when crawling ? For example, we don't want to crawl the following: ^s\.ytim

blacklist for crawling

2010-01-26 Thread Ted Yu
Hi, Is there feature that would enable blacklisting when crawling ? For example, we don't want to crawl the following: ^s\.ytimg\.com/safebrowsing Thanks