> > Why htdig seems to ignore robots.txt entries? > > ht://Dig follows the Standard for Robots Exclusions. You may find useful > these links: > > http://www.it.htdig.org/FAQ.html#q4.22 > http://www.it.htdig.org/attrs.html#robotstxt_name
I got robots.txt like this: User-agent: * Disallow: /page This should disallow all robots index all pages within /page. Right? Besides I have defines robotstxt_name and htdig seems to be visible like ht://Dig with my mail. Piotras ------------------------------------------------------- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
