Interesting to know. The other thing I'm thinking about is do I need to track each request? For DoS checks (back to original subject!) ideally you would track each request, but it could be argued that DoS checks in webserver are only of limited use anyway (as a real DoS could attack the webserver itself), so I'm more worried about protecting only certain URLs that give up dynamic content via my app server and therefore that are more vulnerable to DoS attacks.
So I'm thinking of the following: Move the initcol rules for the IP config to phase 2 (and also update the modsecurity_crs_11_dos_protection.conf phase 1 rules from phase 2), plus have some "ignore static content" rules which "allow" static content at the end of phase 1, thus skipping phase 2. That should dramatically reduce the number of IP addresses tracked in the collection and should hopefully I see less errors. The reason I'm doing the "allow" at the end of phase 1 is so that basic checks (e.g. protocol anomolies, bad robots...etc) are still followed in phase 1. And I'll also need to use the ctl:ruleEngine=On setting as part of my rule to force the use fo "allow" during DetectionOnly mode. Thoughts on this? Obviously this is deviating from the DoS rules as they were intended in the CRS and think I'll probably take a copy of the rules in modsecurity_crs_11_dos_protection.conf rather than directly edit them. Thanks, Barry ________________________________ > From: csand...@trustwave.com > To: jam...@owasp.org; barry_poll...@hotmail.com > CC: owasp-modsecurity-core-rule-set@lists.owasp.org > Subject: Re: [Owasp-modsecurity-core-rule-set] Questions about > modsecurity_crs_11_dos_protection.conf > Date: Wed, 11 Mar 2015 12:13:06 +0000 > > It should be noted for the purposes of this discussion that there is an > experimental build of mod security with memcache support > (https://github.com/SpiderLabs/ModSecurity/tree/memcache_collections). > > From: Josh Amishav-Zlatin <jam...@owasp.org<mailto:jam...@owasp.org>> > Date: Wednesday, March 11, 2015 at 4:37 AM > To: Barry Pollard > <barry_poll...@hotmail.com<mailto:barry_poll...@hotmail.com>> > Cc: Chaim Sanders > <csand...@trustwave.com<mailto:csand...@trustwave.com>>, > "owasp-modsecurity-core-rule-set@lists.owasp.org<mailto:owasp-modsecurity-core-rule-set@lists.owasp.org>" > > <owasp-modsecurity-core-rule-set@lists.owasp.org<mailto:owasp-modsecurity-core-rule-set@lists.owasp.org>> > > Subject: Re: [Owasp-modsecurity-core-rule-set] Questions about > modsecurity_crs_11_dos_protection.conf > > Hi Barry, > > FWIW, we've run into similar issues and re-implemented some rules to > use Redis for storage instead. This had to be implemented via Lua > scripting though, which takes a slight performance hit. Having said > that, there is an open ticket to implement an alternative shared > persistent storage mechanism: > > https://github.com/SpiderLabs/ModSecurity/issues/378<http://scanmail.trustwave.com/?c=4062&d=zP7_1EIfkGQmqzNdTgH9sZIg1Bsu-kZ-wPR_Yvt2BA&s=5&u=https://github.com/SpiderLabs/ModSecurity/issues/378> > > > -- > - Josh > > On Wed, Mar 11, 2015 at 9:49 AM, Barry Pollard > <barry_poll...@hotmail.com<mailto:barry_poll...@hotmail.com>> wrote: > Thanks for those answers Chaim. Make sense to me. > > I do think the use of SDBM for storage for large scale tracking like > for IP addresses is a problem (with ModSecurity rather than with the > core rule set but not that that matters). Unfortunately it just doesn't > work for even relatively small amounts of traffic. > > Not sure how you could resolve it and I know nothing about ModSecurity > is written, or about writing really high performing systems like this, > but came up with below suggestions after a bit of a think. Not sure how > feasible any of these are. > > * Split out IP.pag into several files based on hashed IP address so > bottle neck is not on one file. Would need to be hashed IP address in > case your customer based is skewed towards certain IP ranges. Could > also configure number of files to split across, so busier sites have > larger number of IP.pag files, if the hashing algorithm could take that > config number into account. > > * Have a minimum update time e.g. If hit from IP in last 1 second then > don't bother saving another hit as record is reasonably up to date. > Again time could be configurable. Would obviously impact any DoS/Brute > force rules since we'd be filtering out hits here. > > * Make it easier to ignore the problem by ignoring certain alerts (e.g. > deleting stale collections fails... etc.) unless in high log level, and > add a safe way of ModSecurity clearing down the file and starting again > at certain times (e.g. daily/weekly) or after it reaches a certain > size. Massive fudge but could be short term solution. > > * Use some other persistent storage. Though I'd imagine, given the > potential rate of data updates for a busy site, this would also be a > problem in whatever we move to. > > * Use in-memory storage - though guess that would raise questions on > memory requirements and how to share between different > threads/processes. > > * Have dedicated process/program to manage collections that all other > threads/processes talk via. Could address some of the issues with in > using in-memory storage raised above. Then again could bring in whole > raft of other problems :-) > > * Store less. Collections do seem to be quite large. Could you have a > minimal set for things like IP address with just last access time/num > hits? For rules that require more things tracked (e.g. Brute force > attempts on a specific URL) they could be tracked in a separate "full" > collection which would have a larger record set, but with less entries. > > * Reduce blocking time in code making the SDBM update. Imagine the code > is probably pretty tight here already but obviously any reduction here > would have benefit. Had a quick look myself (more out of nosiness) but > the C code scared me too much (been a while!). > > * Look into how other modules (e.g. mod_evasive) handle this problem to > see if can learn anything from them. > > Am sure you've probably already considered these and better solutions > but thought I'd suggest them anyway. > > Thanks, > Barry > >> On 11 Mar 2015, at 03:54, Chaim Sanders > <csand...@trustwave.com<mailto:csand...@trustwave.com>> wrote: >> >> we > _______________________________________________ > Owasp-modsecurity-core-rule-set mailing list > Owasp-modsecurity-core-rule-set@lists.owasp.org<mailto:Owasp-modsecurity-core-rule-set@lists.owasp.org> > > https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set<http://scanmail.trustwave.com/?c=4062&d=zP7_1EIfkGQmqzNdTgH9sZIg1Bsu-kZ-wKcqY65yWQ&s=5&u=https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set> > > > > ________________________________ > > This transmission may contain information that is privileged, > confidential, and/or exempt from disclosure under applicable law. If > you are not the intended recipient, you are hereby notified that any > disclosure, copying, distribution, or use of the information contained > herein (including any reliance thereon) is strictly prohibited. If you > received this transmission in error, please immediately contact the > sender and destroy the material in its entirety, whether in electronic > or hard copy format. _______________________________________________ Owasp-modsecurity-core-rule-set mailing list Owasp-modsecurity-core-rule-set@lists.owasp.org https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
