On Sat, 14 Apr 2001, shubhendu wrote:
|u r right , actually robots.txt makes it possible that it prevents
|down loads from tools like wget so there will be something in apache
|or whatever webserver the site ( IIRC in my case it was tamacom.com )
|is using to make a differance in calls comming from browser and
|wget
No apache can't waste precious resource in identifying each client that
connects to it and reacting differently... as was pointed out before ..
the robots.txt is just a request not to spider certain parts of the site.
So if you are a decent guy who lives by the rules you wouldn't go to
places where the webmaster doesn't want you to.
|btw log file of wget shows that its using http request on port
|80
the default port for http is 80 so wget connects to that port by default.
|any comments
In ur prev. mail you wanted to know howto prevent access to certain
areas.. use .htaccess and .htpasswd files... so spiders and tools like
wget will not be able to access those areas without proper authentication.
Kingsly
.:: Kingsly John ICQ 14787510 ::.
--------------------------------------------------
.:: Linux 2.4.3 #5 Sun Apr 8 17:10:55 IST 2001 i686 ::.
--------------------------------------------------------
`:. Posted to the list on Sun Apr 15 02:45:35 IST 2001 .:'
----------------------------------------------
LIH is all for free speech. But it was created
for a purpose. Violations of the rules of
this list will result in stern action.
