Thanks ... just the info I needed. ><> ... Jack Community Music Festival - Nov 14 For info see http://docs.google.com/fileview?id=0B2zjKrMU1HCRMTkwOWFlYmYtOTNhMi00NzdlLTk4Y2UtOWNiMWQ3N2UyYmNl&hl=en
Jonathan Swift<http://www.brainyquote.com/quotes/authors/j/jonathan_swift.html> - "May you live every day of your life." On Tue, Nov 10, 2009 at 12:08 PM, Andrew Farnsworth <[email protected]>wrote: > On Tue, Nov 10, 2009 at 12:55 PM, Jack <[email protected]> wrote: > >> What is the 'right way' to snag a copy of the robots.txt file from a web >> site? >> I know search enginges do it all the time before they search a site, so >> could I get it (if it is there by) >> >> telnet <sitename> 80 >> get /robots.txt >> >> or what? >> >> ><> ... Jack >> >> > http://www.microsoft.com/robots.txt > > That will retrieve it, just like yours will. Depends on how you are going > about it, but yes, via their web server. > > Andy > > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en -~----------~----~----~----~------~----~------~--~---
