URL scanning by bots

2013-04-30 Thread André Warnier
Dear Apache developers, This is a suggestion relative to the code of the Apache httpd webserver, and a possible default new default option in the standard distribution of Apache httpd. It also touches on WWW security, which is why I felt that it belongs on this list, rather than on the general

Re: URL scanning by bots

2013-04-30 Thread Reindl Harald
Am 30.04.2013 12:03, schrieb André Warnier: As a general idea thus, anything which impacts the delay to obtain a 404 response, should impact these bots much more than it impacts legitimate users/clients. How much ? Let us imagine for a moment that this suggestion is implemented in the

Re: URL scanning by bots

2013-04-30 Thread Graham Leggett
On 30 Apr 2013, at 12:03 PM, André Warnier a...@ice-sa.com wrote: The only cost would a relatively small change to the Apache webservers, which is what my suggestion consists of : adding a variable delay (say between 100 ms and 2000 ms) to any 404 response. This would have no real effect.

Re: URL scanning by bots

2013-04-30 Thread Christian Folini
Hey André, I do not think your protection mechanism is very good (for reasons mentioned before) But you can try it out for yourself easily with 2-3 ModSecurity rules and the pause directive. Regs, Christian On Tue, Apr 30, 2013 at 12:03:28PM +0200, André Warnier wrote: Dear Apache

Re: URL scanning by bots

2013-04-30 Thread Yehuda Katz
On Tuesday, April 30, 2013, Christian Folini wrote: But you can try it out for yourself easily with 2-3 ModSecurity rules and the pause directive. Someone suggested the same idea to me and I tried it out on one of my servers by setting PHP as the 404 handler and having it loop there. (which

Re: svn commit: r1477652 - in /httpd/httpd/branches/2.4.x: ./ build/ docs/manual/ docs/manual/howto/ docs/manual/mod/ docs/manual/rewrite/ modules/cache/ os/win32/

2013-04-30 Thread Graham Leggett
On 30 Apr 2013, at 4:23 PM, j...@apache.org wrote: Modified: httpd/httpd/branches/2.4.x/ (props changed) httpd/httpd/branches/2.4.x/Apache-apr2.dsw httpd/httpd/branches/2.4.x/Apache.dsw httpd/httpd/branches/2.4.x/CHANGES httpd/httpd/branches/2.4.x/Makefile.win

Re: URL scanning by bots

2013-04-30 Thread Ben Reser
On Tue, Apr 30, 2013 at 3:03 AM, André Warnier a...@ice-sa.com wrote: Let us imagine for a moment that this suggestion is implemented in the Apache webservers, and is enabled in the default configuration. And let's imagine that after a while, 20% of the Apache webservers deployed on the

Re: URL scanning by bots

2013-04-30 Thread Ben Laurie
On 30 April 2013 11:14, Reindl Harald h.rei...@thelounge.net wrote: Am 30.04.2013 12:03, schrieb André Warnier: As a general idea thus, anything which impacts the delay to obtain a 404 response, should impact these bots much more than it impacts legitimate users/clients. How much ? Let us

Re: URL scanning by bots

2013-04-30 Thread Ben Laurie
On 30 April 2013 11:29, Graham Leggett minf...@sharp.fm wrote: On 30 Apr 2013, at 12:03 PM, André Warnier a...@ice-sa.com wrote: The only cost would a relatively small change to the Apache webservers, which is what my suggestion consists of : adding a variable delay (say between 100 ms and

Re: URL scanning by bots

2013-04-30 Thread Graham Leggett
On 30 Apr 2013, at 8:42 PM, Ben Laurie b...@links.org wrote: This would have no real effect. Bots are patient, slowing them down isn't going to inconvenience a bot in any way. The simple workaround if the bot does take too long is to simply send the requests in parallel. Disagree.

Re: URL scanning by bots

2013-04-30 Thread Lazy
2013/4/30 Graham Leggett minf...@sharp.fm On 30 Apr 2013, at 12:03 PM, André Warnier a...@ice-sa.com wrote: The only cost would a relatively small change to the Apache webservers, which is what my suggestion consists of : adding a variable delay (say between 100 ms and 2000 ms) to any

Re: URL scanning by bots

2013-04-30 Thread Steinar H. Gunderson
On Tue, Apr 30, 2013 at 08:54:47PM +0200, Lazy wrote: mod_security + simple scripts+ ipset + iptables TARPIT in the raw table this way You would be able to block efficiently a very large number of ipnumbers, using TARPIT will take care of the delaying new bot connections at minimal cost

Re: URL scanning by bots

2013-04-30 Thread Reindl Harald
Am 30.04.2013 20:38, schrieb Ben Laurie: On 30 April 2013 11:14, Reindl Harald h.rei...@thelounge.net wrote: no - this idea is very very bad and if you ever saw a DDOS-attack from 10 thousands of ip-addresses on a machine you maintain you would not consider anything which makes responses

Re: URL scanning by bots

2013-04-30 Thread André Warnier
Graham Leggett wrote: On 30 Apr 2013, at 12:03 PM, André Warnier a...@ice-sa.com wrote: The only cost would a relatively small change to the Apache webservers, which is what my suggestion consists of : adding a variable delay (say between 100 ms and 2000 ms) to any 404 response. This would

Re: URL scanning by bots

2013-04-30 Thread Ben Reser
On Tue, Apr 30, 2013 at 4:09 PM, André Warnier a...@ice-sa.com wrote: But I have been trying to figure out a real use case, where expecting 404 responses in the course of legitimate applications or website access would be a normal thing to do, and I admit that I haven't been able to think of

Re: URL scanning by bots

2013-04-30 Thread André Warnier
Ben Reser wrote: On Tue, Apr 30, 2013 at 3:03 AM, André Warnier a...@ice-sa.com wrote: Let us imagine for a moment that this suggestion is implemented in the Apache webservers, and is enabled in the default configuration. And let's imagine that after a while, 20% of the Apache webservers

Re: URL scanning by bots

2013-04-30 Thread André Warnier
Ben Reser wrote: On Tue, Apr 30, 2013 at 4:09 PM, André Warnier a...@ice-sa.com wrote: But I have been trying to figure out a real use case, where expecting 404 responses in the course of legitimate applications or website access would be a normal thing to do, and I admit that I haven't been

Re: URL scanning by bots

2013-04-30 Thread André Warnier
Ben Laurie wrote: On 30 April 2013 11:29, Graham Leggett minf...@sharp.fm wrote: On 30 Apr 2013, at 12:03 PM, André Warnier a...@ice-sa.com wrote: The only cost would a relatively small change to the Apache webservers, which is what my suggestion consists of : adding a variable delay (say

Re: URL scanning by bots

2013-04-30 Thread André Warnier
Christian Folini wrote: Hey André, I do not think your protection mechanism is very good (for reasons mentioned before) But you can try it out for yourself easily with 2-3 ModSecurity rules and the pause directive. Regs, Christian Hi Christian. With respect, I think that you