Hi,

I'm doing some research on what technologies various crawlers support for crawl exclusion. Without installing and figuring out Nutch, I can't figure out if it supports the x-robots-tag HTTP header?

Does anybody know?

Thanks,

Mike

Reply via email to