Hello,
I researched this more and never found any small modification to
modsecurity_crs_11_dos_protection.conf that would fix this problem, so I
decided to rewrite the rule.
I pasted in the rule I'm using at the end of this email. I'm far from a
mod_security expert, but I've been running this rule for a few months now
and it seems to be working well in blocking most illegitimate clients while
letting through virtually all legitimate ones. I haven't actually been
attacked, as far as I know of, by anyone intent on actually bringing down
the site, so I'm not sure that it would work under extreme pressure or not.
I'd be very interested in hearing if other people tried it out, or made
modifications to it, as I know it can be improved.
The rule is based on the modsecurity_crs_11_dos_protection.conf rule, along
with some examples I read in a book. There are basically two separate parts
to the rule, and only one or the other needs to be enabled if desired (but
I use both as they work differently).
ID 2106 involves blocking an IP if it requests too quickly, which is
similar to the broken DoS rule but does it a different way using the
update_rate -- this has limitations compared to the theoretical CRS DoS
rule, but doesn't have that bug, so it works much better for me.
Rule 2105 actually keeps track of how much CPU time one IP is using and can
block based on that. So that's more useful if you have some scripts that
are "heavier" than other pages. I have notes in the code but the main weird
thing about that rule is that mod_security 2.6.x (which is what I have)
uses duration in milliseconds and they changed it in 2.7.x to use
microseconds, so if you have 2.7.x you need to multiply by 1000 in order to
get the same result. If using this rule it is very important to adjust that
based on the version of mod_security you are using, or else it's not going
to work.
Rule 2104 adds a delay to fast requests, if they haven't met the block
threshold. I added this after I found that some legitimate bots (like Bing)
sometimes make a ton of requests simultaneously, and I think adding this
delay gets them to slow down the requests a little. I've found that when
using it, I have almost no legitimate clients trigger a block.
And note that this rule probably doesn't play well with the CRS's DoS rule
so you'd want to disable that one. If you do play with it and find a
problem or make an improvement, please let me know! Thanks,
Nick
------
#
# Anti-Automation rule set for detecting Denial of Service Attacks.
#
# This ruleset is designed to monitor both how much server time an
# IP address uses and also how quickly an IP makes requests, and
# can block based on either factor.
#
# RULE NOTE -- IP.duration is in milliseconds in mod_security 2.6.x.
# In mod_security 2.7.x this will change to microseconds, which requires
that
# all these constant factors be multiplied by 1000 to stay the same
#
# Settings of 30000 ms load until block and 250/1 deprecate factor
effectively
# mean that with a 4 core machine, a client can burst up to using one whole
# core for 30s, or 4 cores for 7.5s, before being blocked. If it doesn't use
# a full burst but still is requesting quickly, the 250/1 means that for
every
# 1 second that elapses we'll subtract 250 ms of load, so over time a client
# can use about 1 full core without triggering the ban. This may be too much
# but no legitimate client should use that much so it should be safe.
#
# Enforce an existing IP address block and log only 1-time/minute
# We don't want to get flooded by alerts during an attack or scan so
# we are only triggering an alert once/minute. You can adjust how often
# you want to receive status alerts by changing the expirevar setting below.
#
SecRule IP:DOS_BLOCK "@eq 1" "chain,phase:1,id:'2100',drop,msg:'Denial of
Service (DoS) Attack Identified from %{remote_addr}
(%{tx.dos_block_counter} hits since last
alert)',setvar:ip.dos_block_counter=+1"
SecRule &IP:DOS_BLOCK_FLAG "@eq 0"
"setvar:ip.dos_block_flag=1,expirevar:ip.dos_block_flag=60,setvar:tx.dos_block_counter=%{ip.dos_block_counter},setvar:ip.dos_block_counter=0"
#
# Block and track # of requests but don't log
SecRule IP:DOS_BLOCK "@eq 1"
"phase:1,id:'2101',t:none,drop,nolog,setvar:ip.dos_block_counter=+1"
#
# skipAfter Check
# There are different scenarios where we don't want to do checks -
# 1. If the current IP address has already been blocked due to high requests
# In this case, we skip doing the request counts.
#
SecRule IP:DOS_BLOCK "@eq 1"
"phase:5,id:'2102',t:none,nolog,pass,skipAfter:END_DOS_PROTECTION_CHECKS"
# Keep track of how much web server time is consumed by each IP address
# Decrease the load time by 250 ms every second that elapses so the load
time decreases if the bot
# lightens up on the requests.
SecRule REQUEST_BASENAME "!\.(jpe?g|png|gif|js|css|ico)$"
"phase:5,id:'2103',t:none,nolog,pass,setvar:IP.load=+%{DURATION},deprecatevar:IP.load=250/1"
# If a bot is using a lot of time but isn't actually over the block
threshold yet, slow it down.
# This is most useful for "friendly" bots like bingbot that are just being
more aggressive than
# we'd like. If a bot is trying to do a denial-of-service attack, this
could potentially be
# bad as this Apache process will just sit idle for the delay period. But
I'm not sure that's
# all bad as we have a lot more processes than we have cores, so if this
one is idle then
# another thread will be able to use the CPU time if there is a lot going
on with the server
SecRule IP:LOAD "@ge 15000"
"phase:5,id:'2104',t:none,log,pass,msg:'Delaying aggressive IP
%{remote_addr} - IP load too high: %{IP.load}',pause:5000"
# Block an IP address if it uses too much of the web server's time
SecRule IP:LOAD "@ge 30000"
"phase:5,id:'2105',t:none,log,pass,msg:'Potential Denial of Service (DoS)
Attack from %{remote_addr} - IP load too high:
%{IP.load}',setvar:ip.dos_block=1,expirevar:ip.dos_block=%{tx.dos_block_timeout}"
# Also block an IP address if it makes too many requests too quickly.
# IP.UPDATE_RATE contains the average number of requests per MINUTE, over
the lifetime of the ip collection.
# Note that since it is calculated over its lifetime, if the IP has been
continually requesting pages for
# a long time and then all of a sudden ramps things up a lot, the average
will rise very slowly. This is
# more useful when they start out hitting consistently hard.
# IP.UPDATE_COUNTER is used to make sure it doesn't check the UPDATE_RATE
if only a small number of
# requests have been made, because if an IP makes 5 requests in the same
second we don't necessarily want
# to block immediately if those are all the requests they've made.
SecRule IP:UPDATE_COUNTER "@ge 50"
"chain,phase:5,id:'2106',t:none,log,pass,msg:'Potential Denial of Service
(DoS) Attack from %{remote_addr} - Request update rate: %{IP.UPDATE_RATE}
too high'"
SecRule IP:UPDATE_RATE "@ge 650"
"setvar:ip.dos_block=1,expirevar:ip.dos_block=%{tx.dos_block_timeout}"
SecMarker END_DOS_PROTECTION_CHECKS
------
On Thu, Sep 19, 2013 at 5:31 AM, Slobodan Aleksić <li...@aleksic.de> wrote:
> Hello list,
>
> we have the same problem as Nick mentioned.
> We have loadbalancers in front of Apache reverse proxies and when
> activating the modsecurity_crs_11_dos_protection.conf the loadbalancers
> are causing the rule to trigger.
> We counted the requests - *only* about 120req/hour which is quiet normal.
> A whitelist is helping us, but what is with legitime requests like Nick
> mentioned? Any ideas or an improved version of
> modsecurity_crs_11_dos_protection.conf on the way ?
>
>
> Best regards
>
>
>
>
> On 05/07/2013 07:36 PM, Nick wrote:
> > I've been experimenting with this modsecurity_crs_11_dos_protection.conf
> > ruleset more. An outline of how it seems to work:
> >
> > - Every non-image request increments the ip.dos_counter counter.
> > - If that counter is > tx.dos_counter_threshold, it increments
> > ip.dos_burst_counter, sets an expiration time on ip.dos_burst_counter,
> > and clears ip.dos_counter
> > - If ip.dos_burst_counter > ip.dos_burst_threshold, it blocks the
> > request until tx.dos_block_timeout seconds elapses
> >
> > The rational behind the burst counter seems to be because mod_security
> > expiration doesn't work based on the FIRST write to a variable, it works
> > based on the most recent write. In my case, I'm using a time slice of 60
> > seconds, but if a bot requests a page every 10 seconds for hours it will
> > never expire ip_dos_counter, the only time that will happen is if they
> > don't do any requests for 60 seconds. So, this burst counter is designed
> > to try to get around this problem by waiting until the dos_counter is
> > over the threshold (which doesn't really prove anything about how fast
> > requests were being made, only that the user has done a bunch of
> > requests without pausing) and then it increments the burst counter which
> > has an expiration time (in my case, 60 seconds) and clears out the
> > dos_counter. So, if the dos counter gets up to the threshold AGAIN, then
> > it knows that those requests really all have come in the past 60
> > seconds, so it knows that the user is requesting too quickly.
> >
> > Great in theory, but this is broken.
> >
> > The problem scenario:
> >
> > - A bot (bingbot in my case) has been requesting pages for the past
> > hour. Averaging one or two requests per second. Nothing to worry about.
> > - The dos_counter is over the threshold because it has been submitting
> > requests for a long time without pausing
> > - Sometimes it submits 2-4 requests virtually simultaneously (on the
> > same second in the log)
> > - It submits 2 or more requests simultaneously. For EACH REQUEST,
> > mod_security sees that ip_dos_counter is over the threshold, so each
> > request increments ip_dos_burst_counter. bingbot is now blocked.
> >
> > So this is an example where the bot was blocked even though it was
> > operating at a far slower rate than the DoS settings specify. If those 2
> > requests would have been spaced apart by a second, it would not have
> > been blocked. But, by submitting requests very close together, a bot can
> > be blocked without really submitting that many requests very quickly.
> > This is happening on my server at least every day.
> >
> > I am new to mod_security so I'm still learning exactly how these rules
> > work, but are there any solutions to this problem? The way this rule is
> > set up, it seems to be operating on the principle that the rule is
> > atomic when it clearly isn't. If dos_counter would be cleared before the
> > burst counter is incremented, this problem wouldn't exist.
> >
> > I'm also unclear on how this is coordinated across the multiple apache
> > processes. I understand this to be a collection, which resides on disk,
> > so I'm not sure when mod_security writes this information to disk.
> >
> > Does anyone have any pointers for where to look, or a different DoS
> > ruleset to try?
> >
> > Or is DoS protection not something that is possible with mod_security,
> > and I need another module? I don't need anything fancy, I just want to
> > be able to limit an IP address to a certain number of requests within a
> > time period. Thanks,
> >
> > Nick
> >
> >
> > On Mon, Apr 29, 2013 at 12:49 PM, Nick <darknovan...@gmail.com
> > <mailto:darknovan...@gmail.com>> wrote:
> >
> > Hello,
> >
> > I've been setting up mod_security and enabled
> > the modsecurity_crs_11_dos_protection.conf rule. This is
> > mod_security 2.6.8 and CRS version 2.2.5.
> >
> > I have initialized the settings with:
> > SecAction \
> > "id:'900015', \
> > phase:1, \
> > t:none, \
> > setvar:'tx.dos_burst_time_slice=60', \
> > setvar:'tx.dos_counter_threshold=300', \
> > setvar:'tx.dos_block_timeout=600', \
> > nolog, \
> > pass"
> >
> > This works and it is blocking some very aggressive bots the way it
> > should. But there is a problem. I have occasionally been getting
> > lines like this in the log:
> >
> > Warning. Operator GE matched 2 at IP:dos_burst_counter. [file
> >
> "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_11_dos_protection.conf"]
> > [line "44"] [id "981049"] [msg "Potential Denial of Service (DoS)
> > Attack from 65.55.24.236 - # of Request Bursts: 3"]
> >
> > This bot was actually bingbot. I am new to mod_security, but my
> > understanding of my settings is that it shouldn't block until a bot
> > has requested 300 pages in 60 seconds.
> >
> > When I check the logs I see that IP 65.55.24.236 has requested 313
> > pages in 1 hour. In the 60 seconds before the DoS block happening,
> > this IP only requested 6 pages. This block obviously shouldn't be
> > happening.
> >
> > I am grossly misunderstanding something, or what can I do to fix
> > this? Thanks,
> >
> > Nick
> >
> >
> >
> >
> > This body part will be downloaded on demand.
>
>
_______________________________________________
Owasp-modsecurity-core-rule-set mailing list
Owasp-modsecurity-core-rule-set@lists.owasp.org
https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
