Do FTP sites get spidered as well as web sites ?
Thanks in advance
Steve
---------- Original Message ----------------------------------
From: hostmaster <[EMAIL PROTECTED]>
Date: Mon, 30 Aug 1999 02:54:19 -0500
>Aside from a few bytes of bandwidth, no, none at all. In fact, most of the
>sites on your web server are probably being spidered. You can do some
>reading on using the robots.txt file to prevent it, in the event you have
>content that you would prefer didn't show up in altavista.
>Here's a doc you may find interesting:
>http://www.altavista.com/av/content/addurl_exclude.htm
>
>Best Regards,
>
>Jonathan
>
>At 10:09 AM 8/30/99 +0300, you wrote:
>>Aha!
>>sorry for my naive question but "being spidered" does it cause any
>>problems to the functionality of Server?
>>>The Scooter agent is Altavista's spider. Your web interface (or at least
>>>the front page, as it doesn't have a logon) is being spidered. You'll
>>>also notice that it's coming from DEC.com, which is where Scooter runs from.
>>>
>>>Jonathan
>>>
>>>At 09:17 AM 8/30/99 +0300, you wrote:
>>>>Hi guys!
>>>>
>>>>I was looking the Web Access logs (W2XXXX.log files) and then I noticed
>>>>that the following record appeared sometimes in regural basis....
>>>>
>>>>.......
>>>>19990828 014703 204.123.9.102, Scooter/2.0 G.R.A.B. V1.1.0, , GET /HTTP/1.0
>>>>19990828 015621 204.123.9.102, Scooter/2.0 G.R.A.B. V1.1.0, , GET /HTTP/1.0
>>>>19990828 015649 204.123.9.102, Scooter/2.0 G.R.A.B. V1.1.0, , GET /HTTP/1.0
>>>>19990828 022550 204.123.9.102, Scooter/2.0 G.R.A.B. V1.1.0, , GET /HTTP/1.0
>>>>............
>>>>
>>>>Dou you know what is this???
>>>>
>>>>
>>>>thanks in advance
>>>>Periklis Drakos
>>>>Sys Admin
>>>>School of Social Sciences
>>>>Univ. of Crete, Greece
>
>
>
Please visit http://www.ipswitch.com/support/mailing-lists.html
to be removed from this list.
Please visit http://www.ipswitch.com/support/mailing-lists.html
to be removed from this list.
