On Fri, 13 Apr 2001, shubhendu wrote:
|what is the exact use of this robots.txt file ,
It is used to block access to certain parts of a site when it's being
spidered by a search engine or any automated software.
|what i did i searched for robots.txt in /etc/wgetrc file and found
|that robots = on
It's good netiquette to obey the robots.txt file... because the only
reason the file is there is because the admin doesn't want you to get into
that area except with a browser.
|experimently made it off and tried again this time this file had
|no effect and i got the site
You overrode the robots.txt file... so you got the stuff you wanted.
Kingsly
.:: Kingsly John ICQ 14787510 ::.
--------------------------------------------------
.:: Linux 2.4.3 #5 Sun Apr 8 17:10:55 IST 2001 i686 ::.
--------------------------------------------------------
`:. Posted to the list on Fri Apr 13 16:02:07 IST 2001 .:'
----------------------------------------------
LIH is all for free speech. But it was created
for a purpose. Violations of the rules of
this list will result in stern action.
