There is currently no built-in support for the x-robots-tag header. On Sunday 22 January 2012 01:01:26 Michael Lissner wrote: > Hi, > > I'm doing some research on what technologies various crawlers support > for crawl exclusion. Without installing and figuring out Nutch, I can't > figure out if it supports the x-robots-tag HTTP header? > > Does anybody know? > > Thanks, > > Mike
-- Markus Jelsma - CTO - Openindex
