Hello list, we have the same problem as Nick mentioned. We have loadbalancers in front of Apache reverse proxies and when activating the modsecurity_crs_11_dos_protection.conf the loadbalancers are causing the rule to trigger. We counted the requests - *only* about 120req/hour which is quiet normal. A whitelist is helping us, but what is with legitime requests like Nick mentioned? Any ideas or an improved version of modsecurity_crs_11_dos_protection.conf on the way ?
Best regards On 05/07/2013 07:36 PM, Nick wrote: > I've been experimenting with this modsecurity_crs_11_dos_protection.conf > ruleset more. An outline of how it seems to work: > > - Every non-image request increments the ip.dos_counter counter. > - If that counter is > tx.dos_counter_threshold, it increments > ip.dos_burst_counter, sets an expiration time on ip.dos_burst_counter, > and clears ip.dos_counter > - If ip.dos_burst_counter > ip.dos_burst_threshold, it blocks the > request until tx.dos_block_timeout seconds elapses > > The rational behind the burst counter seems to be because mod_security > expiration doesn't work based on the FIRST write to a variable, it works > based on the most recent write. In my case, I'm using a time slice of 60 > seconds, but if a bot requests a page every 10 seconds for hours it will > never expire ip_dos_counter, the only time that will happen is if they > don't do any requests for 60 seconds. So, this burst counter is designed > to try to get around this problem by waiting until the dos_counter is > over the threshold (which doesn't really prove anything about how fast > requests were being made, only that the user has done a bunch of > requests without pausing) and then it increments the burst counter which > has an expiration time (in my case, 60 seconds) and clears out the > dos_counter. So, if the dos counter gets up to the threshold AGAIN, then > it knows that those requests really all have come in the past 60 > seconds, so it knows that the user is requesting too quickly. > > Great in theory, but this is broken. > > The problem scenario: > > - A bot (bingbot in my case) has been requesting pages for the past > hour. Averaging one or two requests per second. Nothing to worry about. > - The dos_counter is over the threshold because it has been submitting > requests for a long time without pausing > - Sometimes it submits 2-4 requests virtually simultaneously (on the > same second in the log) > - It submits 2 or more requests simultaneously. For EACH REQUEST, > mod_security sees that ip_dos_counter is over the threshold, so each > request increments ip_dos_burst_counter. bingbot is now blocked. > > So this is an example where the bot was blocked even though it was > operating at a far slower rate than the DoS settings specify. If those 2 > requests would have been spaced apart by a second, it would not have > been blocked. But, by submitting requests very close together, a bot can > be blocked without really submitting that many requests very quickly. > This is happening on my server at least every day. > > I am new to mod_security so I'm still learning exactly how these rules > work, but are there any solutions to this problem? The way this rule is > set up, it seems to be operating on the principle that the rule is > atomic when it clearly isn't. If dos_counter would be cleared before the > burst counter is incremented, this problem wouldn't exist. > > I'm also unclear on how this is coordinated across the multiple apache > processes. I understand this to be a collection, which resides on disk, > so I'm not sure when mod_security writes this information to disk. > > Does anyone have any pointers for where to look, or a different DoS > ruleset to try? > > Or is DoS protection not something that is possible with mod_security, > and I need another module? I don't need anything fancy, I just want to > be able to limit an IP address to a certain number of requests within a > time period. Thanks, > > Nick > > > On Mon, Apr 29, 2013 at 12:49 PM, Nick <darknovan...@gmail.com > <mailto:darknovan...@gmail.com>> wrote: > > Hello, > > I've been setting up mod_security and enabled > the modsecurity_crs_11_dos_protection.conf rule. This is > mod_security 2.6.8 and CRS version 2.2.5. > > I have initialized the settings with: > SecAction \ > "id:'900015', \ > phase:1, \ > t:none, \ > setvar:'tx.dos_burst_time_slice=60', \ > setvar:'tx.dos_counter_threshold=300', \ > setvar:'tx.dos_block_timeout=600', \ > nolog, \ > pass" > > This works and it is blocking some very aggressive bots the way it > should. But there is a problem. I have occasionally been getting > lines like this in the log: > > Warning. Operator GE matched 2 at IP:dos_burst_counter. [file > > "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_11_dos_protection.conf"] > [line "44"] [id "981049"] [msg "Potential Denial of Service (DoS) > Attack from 65.55.24.236 - # of Request Bursts: 3"] > > This bot was actually bingbot. I am new to mod_security, but my > understanding of my settings is that it shouldn't block until a bot > has requested 300 pages in 60 seconds. > > When I check the logs I see that IP 65.55.24.236 has requested 313 > pages in 1 hour. In the 60 seconds before the DoS block happening, > this IP only requested 6 pages. This block obviously shouldn't be > happening. > > I am grossly misunderstanding something, or what can I do to fix > this? Thanks, > > Nick > > > > > This body part will be downloaded on demand. _______________________________________________ Owasp-modsecurity-core-rule-set mailing list Owasp-modsecurity-core-rule-set@lists.owasp.org https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
