Thanks for the follow up. Very good to know. Seems like this would be
pretty easy to add, given that the robots meta tag is supported.
Are there any plans to do so?
Mike
On Mon 23 Jan 2012 02:27:02 AM PST, Markus Jelsma wrote:
There is currently no built-in support for the x-robots-tag header.
On Sunday 22 January 2012 01:01:26 Michael Lissner wrote:
Hi,
I'm doing some research on what technologies various crawlers support
for crawl exclusion. Without installing and figuring out Nutch, I can't
figure out if it supports the x-robots-tag HTTP header?
Does anybody know?
Thanks,
Mike
