Or if you happen to be using Python ...for some reason they have a standard module that will not only load the file for you, it'll parse it and answer questions about what it says you can fetch.
http://docs.python.org/library/robotparser.html >>> import robotparser >>> rp = robotparser.RobotFileParser() >>> rp.set_url("http://www.musi-cal.com/robots.txt";) >>> rp.read() >>> rp.can_fetch("*", >>> "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco";) False >>> rp.can_fetch("*", "http://www.musi-cal.com/";) True -j --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en -~----------~----~----~----~------~----~------~--~---
