Hello folks, How can i crawl an URL but not index it not store it ? Ie for example I give a list of URL to crawl and I need a method to remove or avoid indexing of some of those URL in order to not pollute the database. Fabrice.
Hello folks, How can i crawl an URL but not index it not store it ? Ie for example I give a list of URL to crawl and I need a method to remove or avoid indexing of some of those URL in order to not pollute the database. Fabrice.