On 5/22/07, Robin Green [EMAIL PROTECTED] wrote:
On Tue, 22 May 2007 15:05:48 +0100
Duncan Coutts [EMAIL PROTECTED] wrote:
On Tue, 2007-05-22 at 14:40 +0100, Claus Reinke wrote:
so the situation for mailing lists and online docs seems to have
improved, but there is still the wiki
as was pointed out on the programming reddit [1], crawling of the
haskell wiki is forbidden, since http://www.haskell.org/robots.txt contains
User-agent: *
Disallow: /haskellwiki/
and indeed, a google search gives the old wiki
http://www.google.ch/search?q=haskell+wiki
i.e.
On Tue, 22 May 2007, Vincent Kraeutler wrote:
as was pointed out on the programming reddit [1], crawling of the
haskell wiki is forbidden, since http://www.haskell.org/robots.txt contains
User-agent: *
Disallow: /haskellwiki/
and indeed, a google search gives the old wiki
as was pointed out on the programming reddit [1], crawling of the
haskell wiki is forbidden, since http://www.haskell.org/robots.txt contains
User-agent: *
Disallow: /haskellwiki/
i agree that having the wiki searchable would be preferred,
but was told that there were performance issues. even
On Tue, 2007-05-22 at 14:40 +0100, Claus Reinke wrote:
so the situation for mailing lists and online docs seems to have
improved, but there is still the wiki indexing/rogue bot issue,
and lots of fine tuning (together with watching the logs to spot
any issues arising out of relaxing those
On Tue, 22 May 2007, Duncan Coutts wrote:
So if we can ban bots from the page histories or turn them off for the
bot user agents or something then we might have a cure. Perhaps we just
need to upgrade our media wiki software or find out how other sites
using this software deal with the same
On Tue, 2007-05-22 at 16:26 +0200, Henning Thielemann wrote:
On Tue, 22 May 2007, Duncan Coutts wrote:
So if we can ban bots from the page histories or turn them off for the
bot user agents or something then we might have a cure. Perhaps we just
need to upgrade our media wiki software or
So if we can ban bots from the page histories or turn
them off for the
bot user agents or something then we might have a cure.
Perhaps we just
need to upgrade our media wiki software or find out how
other sites
using this software deal with the same issue of bots reading page
On Tue, 22 May 2007 15:05:48 +0100
Duncan Coutts [EMAIL PROTECTED] wrote:
On Tue, 2007-05-22 at 14:40 +0100, Claus Reinke wrote:
so the situation for mailing lists and online docs seems to have
improved, but there is still the wiki indexing/rogue bot issue,
and lots of fine tuning
The wiki could be configured to use /haskellwiki/index.php?.. urls for
diffs (I believe this can be done by changing $wgScript). Then
robots.txt could be changed to
Disallow: /haskellwiki/index.php
Which bans robots from everything except normal pages.
that sounds like the most promising
10 matches
Mail list logo
