Re: [LIH] RE:shell account/robots.txt

[Suresh Ramasubramanian]

shubhendu rearranged electrons thusly:

> actually while using wget  for a site i found that this file was getting
> downloades automaitcally and due to denyal of some dirs the data that i
> wanted was not comming down 
 
that's what a robots.txt is for.

> what i did i searched for robots.txt in /etc/wgetrc file and found 
> that robots = on 

correctly so.  robots.txt is the web admin's way of saying "keep off" to
webcrawlers etc.

> so what it is exactly ?????
 
see above (or search google for robots.txt)

-- 
Suresh Ramasubramanian  <-->  mallet <at> efn <dot> org
EMail Sturmbannfuhrer, Lower Middle Class Unix Sysadmin


----------------------------------------------
An alpha version of a web based tool to manage
your subscription with this mailing list is at
http://lists.linux-india.org/cgi-bin/mj_wwwusr