as far I can see nutch's html parser does only support the meta tag noindex (<meta name="ROBOTS" content="NOINDEX,NOFOLLOW"> ) but there is an inoffiziel html <noindex> tag. http://www.webmasterworld.com/forum10003/2703.htm
Hello Stefan, Here is a previous discussion about this : http://www.mail-archive.com/[email protected]/msg04576.html
May be this would be another thing to make nutch more polite.
+1 Jérôme -- http://motrech.free.fr/ http://www.frutch.org/
