I certainly have no objection. I would recommend, however, that the default setting of this configuration option be set to "follow metadata/rel", and that the implementation be backwards compatible.
Thanks, Karl On Sat, Feb 25, 2017 at 5:02 PM, Markus Schuch <[email protected]> wrote: > Hi, > > what do you think about adding the possibility to ignore meta/rel robots > tags/attributes? > > I know, such a thing is an unpolite behavior for a webcrawler, but we > already have the feature to ignore the robots.txt and for me it was > unexpected, when i configured the crawler to ignore robots.txt but it > still respected the meta/rel robots tags/attibutes. > > I will open a ticket, if there are no objections. > > Thanks in advance. > Markus >
