[LIH] RE:shell account/robots.txt

[shubhendu]

hi Suresh

> > Disallow: /cgi/
> 
> > Does that mean, I cannot create dir's with these names ?
>  
>  No.  Someone is clued enough to put up this file so that search engine bots
>  dont go and spider / index those directories.
>  

what is the exact use of this robots.txt file , 

actually while using wget  for a site i found that this file was getting
downloades automaitcally and due to denyal of some dirs the data that i
wanted was not comming down 

what i did i searched for robots.txt in /etc/wgetrc file and found 
that robots = on 

experimently made it off and tried again this time this file had
no effect and i got the site 

so what it is exactly ?????

regards 

shubh
REC-Surat
--------------------------
            @>
i love    /( )\  linux
           ^ ^ 
--------------------------




----------------------------------------------
LIH is all for free speech.  But it was created
for a purpose.  Violations of the rules of
this list will result in stern action.