Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Arjen Markus
On Tue, 30 Oct 2012 06:17:05 -0400 Richard Hipp d...@sqlite.org wrote: This two-phase defense against bots is usually effective. But last night, a couple of bots got through on the SQLite website. No great damage was done as we have ample bandwidth and CPU reserves to handle this sort of

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Lluís Batlle i Rossell
On Tue, Oct 30, 2012 at 06:17:05AM -0400, Richard Hipp wrote: Finally: Do you have any further ideas on how to defend a Fossil website against runs such as the two we observed on SQLite last night? This problem affects almost any web software, and I think that job is delegated to robots.txt.

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Kees Nuyt
On Tue, 30 Oct 2012 06:17:05 -0400, Richard Hipp d...@sqlite.org wrote: [...] Both sessions started out innocently. The logs suggest that there really was a human operator initially. But then after about 3 minutes of normal browsing, each session starts downloading every hyperlink in sight

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Richard Hipp
On Tue, Oct 30, 2012 at 6:23 AM, Lluís Batlle i Rossell vi...@viric.namewrote: On Tue, Oct 30, 2012 at 06:17:05AM -0400, Richard Hipp wrote: Finally: Do you have any further ideas on how to defend a Fossil website against runs such as the two we observed on SQLite last night? This problem

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Bernd Paysan
Am Dienstag, 30. Oktober 2012, 08:20:14 schrieb Richard Hipp: On Tue, Oct 30, 2012 at 6:23 AM, Lluís Batlle i Rossell vi...@viric.namewrote: On Tue, Oct 30, 2012 at 06:17:05AM -0400, Richard Hipp wrote: Finally: Do you have any further ideas on how to defend a Fossil website against runs

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Kees Nuyt
[Default] On Tue, 30 Oct 2012 06:17:05 -0400, Richard Hipp d...@sqlite.org wrote: Finally: Do you have any further ideas on how to defend a Fossil website against runs such as the two we observed on SQLite last night? Another suggestion: Include a (mostly invisible, perhaps hard to recognize)

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Kees Nuyt
[Default] On Tue, 30 Oct 2012 10:13:47 -0500, Nolan Darilek no...@thewordnerd.info wrote: And, most importantly, don't sacrifice accessibility in the name of excluding bots. Mouseover links are notoriously inaccessible. Same with only adding href on focus via JS rather than on page load. If

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Steve Havelka
My guess is that you don't really want to filter out bots, specifically, but really anyone who's attempting to hit every link Fossil makes--that is to say, it's the behavior that we're trying to stop here, not the actor. I suppose what I'd do is set up a mechanism to detect when the remote user