Hi,I'm doing some research on what technologies various crawlers support for crawl exclusion. Without installing and figuring out Nutch, I can't figure out if it supports the x-robots-tag HTTP header?
Does anybody know? Thanks, Mike
Hi,I'm doing some research on what technologies various crawlers support for crawl exclusion. Without installing and figuring out Nutch, I can't figure out if it supports the x-robots-tag HTTP header?
Does anybody know? Thanks, Mike