OK, issue created: https://issues.apache.org/jira/browse/NUTCH-1257
Thanks again.
Mike
On 01/24/2012 11:44 PM, Markus Jelsma wrote:
Not that i'm aware of. You can create an issue in Jira if you like and provide
a patch if possible.
Thanks for the follow up. Very good to know. Seems like this would be
pretty easy to add, given that the robots meta tag is supported.
Are there any plans to do so?
Mike
On Mon 23 Jan 2012 02:27:02 AM PST, Markus Jelsma wrote:
There is currently no built-in support for the x-robots-tag header.
On Sunday 22 January 2012 01:01:26 Michael Lissner wrote:
Hi,
I'm doing some research on what technologies various crawlers support
for crawl exclusion. Without installing and figuring out Nutch, I can't
figure out if it supports the x-robots-tag HTTP header?
Does anybody know?
Thanks,
Mike
