On Wed, Nov 11, 2009 at 8:52 AM, Jason Orendorff <[email protected]> wrote: > > Or if you happen to be using Python ...for some reason they have a > standard module that will not only load the file for you, it'll parse > it and answer questions about what it says you can fetch. > > http://docs.python.org/library/robotparser.html >
Ruby has this too: http://github.com/parolkar/obey_robots_dot_txt/ Andrew --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en -~----------~----~----~----~------~----~------~--~---
