What is the 'right way' to snag a copy of the robots.txt file from a web site? I know search enginges do it all the time before they search a site, so could I get it (if it is there by)
telnet <sitename> 80 get /robots.txt or what? ><> ... Jack Community Music Festival - Nov 14 For info see http://docs.google.com/fileview?id=0B2zjKrMU1HCRMTkwOWFlYmYtOTNhMi00NzdlLTk4Y2UtOWNiMWQ3N2UyYmNl&hl=en Stephen Leacock<http://www.brainyquote.com/quotes/authors/s/stephen_leacock.html> - "I detest life-insurance agents: they always argue that I shall some day die, which is not so." --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en -~----------~----~----~----~------~----~------~--~---
