or use an http library for your language of choice as I am presuming you are writing code to look at web content if you are reading the robots.txt, then parse it.
Looks like this is a good site to read: http://www.robotstxt.org/ and this is a good page on it there: http://www.robotstxt.org/robotstxt.html Andy --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en -~----------~----~----~----~------~----~------~--~---
