Hi, as far I can see nutch's html parser does only support the meta tag noindex (<meta name="ROBOTS" content="NOINDEX,NOFOLLOW"> ) but there is an inoffiziel html <noindex> tag. http://www.webmasterworld.com/forum10003/2703.htm
May be this would be another thing to make nutch more polite. Also please remember my patch to support crawl-delay properties in robots.txt. That would be also something important to make nutch more polite and may be a better way than removing the nutch crawler identification. Thoughts? Stefan Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
