On Tue, Nov 10, 2009 at 12:55 PM, Jack <[email protected]> wrote: > What is the 'right way' to snag a copy of the robots.txt file from a web > site? > I know search enginges do it all the time before they search a site, so > could I get it (if it is there by) > > telnet <sitename> 80 > get /robots.txt > > or what? > > ><> ... Jack > > http://www.microsoft.com/robots.txt
That will retrieve it, just like yours will. Depends on how you are going about it, but yes, via their web server. Andy --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en -~----------~----~----~----~------~----~------~--~---
