Thanks for those answers Chaim. Make sense to me. I do think the use of SDBM for storage for large scale tracking like for IP addresses is a problem (with ModSecurity rather than with the core rule set but not that that matters). Unfortunately it just doesn't work for even relatively small amounts of traffic.
Not sure how you could resolve it and I know nothing about ModSecurity is written, or about writing really high performing systems like this, but came up with below suggestions after a bit of a think. Not sure how feasible any of these are. * Split out IP.pag into several files based on hashed IP address so bottle neck is not on one file. Would need to be hashed IP address in case your customer based is skewed towards certain IP ranges. Could also configure number of files to split across, so busier sites have larger number of IP.pag files, if the hashing algorithm could take that config number into account. * Have a minimum update time e.g. If hit from IP in last 1 second then don't bother saving another hit as record is reasonably up to date. Again time could be configurable. Would obviously impact any DoS/Brute force rules since we'd be filtering out hits here. * Make it easier to ignore the problem by ignoring certain alerts (e.g. deleting stale collections fails... etc.) unless in high log level, and add a safe way of ModSecurity clearing down the file and starting again at certain times (e.g. daily/weekly) or after it reaches a certain size. Massive fudge but could be short term solution. * Use some other persistent storage. Though I'd imagine, given the potential rate of data updates for a busy site, this would also be a problem in whatever we move to. * Use in-memory storage - though guess that would raise questions on memory requirements and how to share between different threads/processes. * Have dedicated process/program to manage collections that all other threads/processes talk via. Could address some of the issues with in using in-memory storage raised above. Then again could bring in whole raft of other problems :-) * Store less. Collections do seem to be quite large. Could you have a minimal set for things like IP address with just last access time/num hits? For rules that require more things tracked (e.g. Brute force attempts on a specific URL) they could be tracked in a separate "full" collection which would have a larger record set, but with less entries. * Reduce blocking time in code making the SDBM update. Imagine the code is probably pretty tight here already but obviously any reduction here would have benefit. Had a quick look myself (more out of nosiness) but the C code scared me too much (been a while!). * Look into how other modules (e.g. mod_evasive) handle this problem to see if can learn anything from them. Am sure you've probably already considered these and better solutions but thought I'd suggest them anyway. Thanks, Barry > On 11 Mar 2015, at 03:54, Chaim Sanders <csand...@trustwave.com> wrote: > > we _______________________________________________ Owasp-modsecurity-core-rule-set mailing list Owasp-modsecurity-core-rule-set@lists.owasp.org https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
