Peter C. Ndikuwera wrote:
I believe Google obeys the robots.txt rule
It does but some robots don't just do sites they look for "classified"
info you can't protect with just a robots.txt.Some searches just go
beyond "boundaries" :-)
I appreciate though
Ronny
http://www.robotstxt.org/wc/norobots.html
Peter
On Mon, 2006-03-20 at 15:20 +0300, Ronny wrote:
Hi all,just a quick question how can one kick out google robots from
crawling some prohibited networks not sites but still access all google
tools am not anti-google ( I use it every hour) but just got a
revelation ;-)
Thanks
Ronny
-----------------------
Random verse of the day:
And who, as I, shall call, and shall declare it, and set it in order for
me, since I appointed the ancient people? and the things that are
coming, and shall come, let them shew unto them. -- Isaiah 44:7
_______________________________________________
LUG mailing list
[email protected]
http://kym.net/mailman/listinfo/lug
%LUG is generously hosted by INFOCOM http://www.infocom.co.ug/
The above comments and data are owned by whoever posted them (including attachments if any). The List's Host is not responsible for them in any way.
---------------------------------------
|
_______________________________________________
LUG mailing list
[email protected]
http://kym.net/mailman/listinfo/lug
%LUG is generously hosted by INFOCOM http://www.infocom.co.ug/
The above comments and data are owned by whoever posted them (including
attachments if any). The List's Host is not responsible for them in any way.
---------------------------------------
