Hi, what do you think about adding the possibility to ignore meta/rel robots tags/attributes?
I know, such a thing is an unpolite behavior for a webcrawler, but we already have the feature to ignore the robots.txt and for me it was unexpected, when i configured the crawler to ignore robots.txt but it still respected the meta/rel robots tags/attibutes. I will open a ticket, if there are no objections. Thanks in advance. Markus
