[fossil-users] Help improve bot exclusion

2012-10-30 Thread Richard Hipp
A Fossil website for a project with a few thousand check-ins can have a lot of hyperlinks. If a spider or bot starts to walk that site, it will visit literally hundreds of thousand or perhaps millions of pages, many of which are things like vdiff and annotate which are computationally expensive

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Arjen Markus
On Tue, 30 Oct 2012 06:17:05 -0400 Richard Hipp d...@sqlite.org wrote: This two-phase defense against bots is usually effective. But last night, a couple of bots got through on the SQLite website. No great damage was done as we have ample bandwidth and CPU reserves to handle this sort of

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Lluís Batlle i Rossell
On Tue, Oct 30, 2012 at 06:17:05AM -0400, Richard Hipp wrote: Finally: Do you have any further ideas on how to defend a Fossil website against runs such as the two we observed on SQLite last night? This problem affects almost any web software, and I think that job is delegated to robots.txt.

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Kees Nuyt
On Tue, 30 Oct 2012 06:17:05 -0400, Richard Hipp d...@sqlite.org wrote: [...] Both sessions started out innocently. The logs suggest that there really was a human operator initially. But then after about 3 minutes of normal browsing, each session starts downloading every hyperlink in sight

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Richard Hipp
On Tue, Oct 30, 2012 at 6:23 AM, Lluís Batlle i Rossell vi...@viric.namewrote: On Tue, Oct 30, 2012 at 06:17:05AM -0400, Richard Hipp wrote: Finally: Do you have any further ideas on how to defend a Fossil website against runs such as the two we observed on SQLite last night? This problem

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Bernd Paysan
Am Dienstag, 30. Oktober 2012, 08:20:14 schrieb Richard Hipp: On Tue, Oct 30, 2012 at 6:23 AM, Lluís Batlle i Rossell vi...@viric.namewrote: On Tue, Oct 30, 2012 at 06:17:05AM -0400, Richard Hipp wrote: Finally: Do you have any further ideas on how to defend a Fossil website against runs

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Kees Nuyt
[Default] On Tue, 30 Oct 2012 06:17:05 -0400, Richard Hipp d...@sqlite.org wrote: Finally: Do you have any further ideas on how to defend a Fossil website against runs such as the two we observed on SQLite last night? Another suggestion: Include a (mostly invisible, perhaps hard to recognize)

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Kees Nuyt
[Default] On Tue, 30 Oct 2012 10:13:47 -0500, Nolan Darilek no...@thewordnerd.info wrote: And, most importantly, don't sacrifice accessibility in the name of excluding bots. Mouseover links are notoriously inaccessible. Same with only adding href on focus via JS rather than on page load. If

Re: [fossil-users] Help improve bot exclusion

2012-10-30 Thread Steve Havelka
My guess is that you don't really want to filter out bots, specifically, but really anyone who's attempting to hit every link Fossil makes--that is to say, it's the behavior that we're trying to stop here, not the actor. I suppose what I'd do is set up a mechanism to detect when the remote user

Re: [fossil-users] sync bug in 1.24

2012-10-30 Thread Richard Hipp
On Tue, Oct 30, 2012 at 12:20 PM, Barak A. Pearlmutter ba...@cs.nuim.iewrote: I installed the pre-compiled fossil 1.24 as /usr/local/bin/fossil and then on my server ran fossil rebuild; fossil serve in the repo directory and fossil rebuild on the client, in the repo. Both on my own copy of

[fossil-users] fossil all rebuild seg faults

2012-10-30 Thread James Turner
With the latest fossil trunk (This is fossil version 1.24 [bdbe6c74b8] 2012-10-30 18:14:27 UTC) fossil all rebuild is seg faulting for me. fossil all rebuild Segmentation fault (core dumped) gdb is showing the below: #0 collect_arguments (zArg=0x7f7f Address 0x7f7f out of

Re: [fossil-users] fossil all rebuild seg faults

2012-10-30 Thread Richard Hipp
Please try the latest and let me know whether or not the problem is fixed. Tnx for the report. On Tue, Oct 30, 2012 at 6:31 PM, James Turner ja...@calminferno.net wrote: With the latest fossil trunk (This is fossil version 1.24 [bdbe6c74b8] 2012-10-30 18:14:27 UTC) fossil all rebuild is seg

Re: [fossil-users] fossil all rebuild seg faults

2012-10-30 Thread James Turner
Looks good. fossil all rebuild is working for me again. If it helps explain anything, I'm running OpenBSD. On Tue, Oct 30, 2012 at 08:11:53PM -0400, Richard Hipp wrote: Please try the latest and let me know whether or not the problem is fixed. Tnx for the report. On Tue, Oct 30, 2012 at 6:31

Re: [fossil-users] fossil all rebuild seg faults

2012-10-30 Thread Steve Bennett
On 31/10/2012, at 10:11 AM, Richard Hipp wrote: Please try the latest and let me know whether or not the problem is fixed. Tnx for the report. Regarding your latest commit, I've run across this on 64 bit too. The problem is the '0' at the end of the variable args. Use NULL instead, otherwise

Re: [fossil-users] sync bug in 1.24

2012-10-30 Thread Richard Hipp
On Tue, Oct 30, 2012 at 6:18 PM, Barak A. Pearlmutter ba...@cs.nuim.iewrote: Is your server running an earlier version of Fossil? As I said, I installed the pre-compiled fossil 1.24 as /usr/local/bin/fossil and then on my server ran fossil rebuild; fossil serve in the repo directory

Re: [fossil-users] fossil all rebuild seg faults

2012-10-30 Thread Richard Hipp
On Tue, Oct 30, 2012 at 8:18 PM, Steve Bennett ste...@workware.net.auwrote: On 31/10/2012, at 10:11 AM, Richard Hipp wrote: Please try the latest and let me know whether or not the problem is fixed. Tnx for the report. Regarding your latest commit, I've run across this on 64 bit too. The

Re: [fossil-users] fossil all rebuild seg faults

2012-10-30 Thread James Turner
On Tue, Oct 30, 2012 at 08:22:25PM -0400, Richard Hipp wrote: On Tue, Oct 30, 2012 at 8:18 PM, Steve Bennett ste...@workware.net.auwrote: On 31/10/2012, at 10:11 AM, Richard Hipp wrote: Please try the latest and let me know whether or not the problem is fixed. Tnx for the report.