...@wustl.edu
>>
>> -Original Message-
>> From: galaxy-dev-boun...@lists.bx.psu.edu
>> [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of SHAUN WEBB
>> Sent: Monday, December 12, 2011 8:40 AM
>> To: galaxy dev
>> Subject: [galaxy-dev] googlebot
>>
galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of SHAUN WEBB
Sent: Monday, December 12, 2011 8:40 AM
To: galaxy dev
Subject: [galaxy-dev] googlebot
Hi,
my paster.log is full of errors relating to a googlebot trying to
access a library file
y dev
Subject: [galaxy-dev] googlebot
Hi,
my paster.log is full of errors relating to a googlebot trying to access a
library file.
I noticed a robots.txt file in the static folder configured as such.
User-agent: *
Disallow: /
I thought this would stop all bots accessing all files. Should th
Hi,
my paster.log is full of errors relating to a googlebot trying to
access a library file.
I noticed a robots.txt file in the static folder configured as such.
User-agent: *
Disallow: /
I thought this would stop all bots accessing all files. Should this be
configured differently?
Tha
