Hello everybody, I'm used to filter urls before fetch operation by using regex-filter to avoid crawling the world wide web.
I've got a specific need: one main page giving all urls to crawl. I want to crawl the main page to have outlinks but I dont want to index this page. How can I proceed? I could enable this feature in my specific plugin but I want to be sure nothing is already existing as ever ;) Dirty solution would be to delete this main page url in the generated solr index with a json query but yeah this is really dirty ;) Hope I'm clear.
