as far I can see nutch's html parser does only support the meta tag
noindex (<meta name="ROBOTS" content="NOINDEX,NOFOLLOW"> ) but there
is an inoffiziel html <noindex> tag.
http://www.webmasterworld.com/forum10003/2703.htm
Hello Stefan,
Here is a previous discussion about this :
http://www.mail-archive.com/[email protected]/msg04576.html
May be this would be another thing to make nutch more polite.
+1
Jérôme
--
http://motrech.free.fr/
http://www.frutch.org/
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
