In addition to Barry’s fantastic links I’ll direct you to this post written by Ryan Barnett that details how to add exceptions in an effective method. There is some tuning required as mentioned before but after about a week these tend to go away pretty quickly and just rarely rear their head. Check it out https://www.trustwave.com/Resources/SpiderLabs-Blog/ModSecurity-Advanced-Topic-of-the-Week--(Updated)-Exception-Handling/. If you need additional help please do reach out ☺
Chaim Sanders Security Researcher Trustwave | SMART SECURITY ON DEMAND www.trustwave.com<http://www.trustwave.com/> From: owasp-modsecurity-core-rule-set-boun...@lists.owasp.org [mailto:owasp-modsecurity-core-rule-set-boun...@lists.owasp.org] On Behalf Of T. Kenneth Lojo (IRRI) Sent: Tuesday, May 17, 2016 3:51 AM To: Barry Pollard <barry_poll...@hotmail.com> Cc: owasp-modsecurity-core-rule-set@lists.owasp.org Subject: Re: [Owasp-modsecurity-core-rule-set] Facebook scrape problem Thank you for this Barry. We actually have attacks going on on the server and so far mod_sec has blocked it with the CRS. Our server with wordpress is being brute forced by a xmlrpc.php attack and our joomla server is being flooded by a torrent request when we don't have any torrents running. I may need to do both in parallel. Do the anomaly scoring to allow facebook scraping the content and probably setup another server and do your recommendation to better tweak the WAF and then apply to the production. On Tue, May 17, 2016 at 3:05 PM, Barry Pollard <barry_poll...@hotmail.com<mailto:barry_poll...@hotmail.com>> wrote: I would say first thing is to turn blocking off and run in DetectionOnly mode to help you fine tune your rules. To do that update your SecRuleEngine config like so: SecRuleEngine DetectionOnly This will of course mean you are not protected but us a necessary step to getting your set up right. Now leave it run for a while and then check the logs for every rule that fired (but did not block this time) and categories them into: 1) False positive - this request looks to be the sort of request our application expects and ModSecurity should not be alerting on it. 2) Bad request - this request shouldn't be made and ModSecurity was right to block it. This will include bots and scripts that scan websites even if they don't cause any trouble. For all the 1s (and there will be a lot at the beginning) you need to decide how to tweak the rules to not alert for False positives. This involves either turning the rule off completely (using the likes of SecRuleRemoveById), turning it off for particular parameters (using the likes of SecRuleUpdateActionById) or turning it off for particular URLs (this is more complicated aged and can require building a new rule to do this). Tuning rules is necessary and, as long as you have a good understanding of what the rules intention is, why it blocked, why that was incorrect thing for it to do then you should tweak and turn them off for certain scenarios. Does that reduce the effectiveness of ModSecurity? Potentially but then if it blocks all sorts of real visitors incorrectly then that's not much use is it! The OWASP CRS is very generic and all the rules will not be appropriate for all websites. By default it blocks too much. There is some work going on to make the default more lenient so you can start off with some protection and ramp up as you see fit rather than current situation where you start with too much protection and have to ramp down. Anyway after you see no more false positives for a while you can turn SecRuleEngine back to on. This can take some time. See my story here to prepare you: http://stackoverflow.com/questions/35149264/how-long-do-you-fine-tune-false-positives-with-mod-security-and-owasp-rules/35162976#35162976<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jfp1DhFSw&s=5&u=http%3a%2f%2fstackoverflow%2ecom%2fquestions%2f35149264%2fhow-long-do-you-fine-tune-false-positives-with-mod-security-and-owasp-rules%2f35162976%2335162976>. A WAF like ModSecurity is, unfortunately, not just a turn it on and it works and you can forget about it solution. It takes a lot of set up to be useful, and then a bit if minding afterwards. Personally I think it's worth it but you also see people online saying WAFs are too much effort for this reason. Anomaly scoring mode is an interesting one. It basically let's all the rules run and then only blocks if a certain threshold applies. This means a number of unimportant rules can fire. e.g. most browsers send a user agent so no user agent, while it likely won't cause a problem is a flag that this is probably a bad request. If a few of these flags fire on same request then this is highly likely to be a bad request and should be blocked. Some rules (like missing user agent) might have low threshold and so won't block on their own and some will block with just one rule firing if it's obvious this request should not processed. While anomaly scoring is undoubtedly helpful to reduce the number of incorrect blocks, and lots of people use it and recommend it, I'm not a particular fan. I find it makes the log files noisy and confusing to see rules firing and not know if they caused a block or not. I prefer to turn off the low value rules completely and using the original block at first bad attempt mode despite the fact this takes extra work to set up initially and can allow more spam and bad bots through. But each to their own. Will leave Christian to explain how best to set it up if that's the way you want to go. Here's some other posts that might help: http://stackoverflow.com/questions/33676348/extra-sensitive-mod-security-rules-giving-403-forbidden-error<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44mTj0m1ERQ&s=5&u=http%3a%2f%2fstackoverflow%2ecom%2fquestions%2f33676348%2fextra-sensitive-mod-security-rules-giving-403-forbidden-error> http://stackoverflow.com/questions/34478019/keep-modsecurity-enabled-with-symfony-installation-w-cpanel-whm/34484463#34484463<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44j6x0GVFTg&s=5&u=http%3a%2f%2fstackoverflow%2ecom%2fquestions%2f34478019%2fkeep-modsecurity-enabled-with-symfony-installation-w-cpanel-whm%2f34484463%2334484463> http://stackoverflow.com/questions/33989273/modsecurity-excessive-false-positives/34027786#34027786<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jGz12xDTw&s=5&u=http%3a%2f%2fstackoverflow%2ecom%2fquestions%2f33989273%2fmodsecurity-excessive-false-positives%2f34027786%2334027786> Note this mailing list is awesome and you will get help here but I have also been answering ModSecurity questions on StackOverflow/ServerFault as feel they are better to reference again for common questions like yours. Been meaning to write a friendly, short, beginners containing a lot of the detail here but have a problem keeping my posts short :-) Hope that helps and feel free to ask any questions here. We're a friendly bunch. Thanks, Barry -- T. Kenneth S. Lojo Specialist-Online Media Design [Image removed by sender. IRRI]<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jay1mwTSg&s=5&u=http%3a%2f%2firri%2eorg%2f> +63 2 580 5600 ext. 2703/2744 +63 928 209 1191 (mobile) t.l...@irri.org<mailto:g.lav...@irri.org> www.irri.org<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jPnhj0STA&s=5&u=http%3a%2f%2fwww%2eirri%2eorg> [Image removed by sender. Facebook]<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44mTggTpFSg&s=5&u=http%3a%2f%2fwww%2efacebook%2ecom%2fIRRI%2ericenews> [Image removed by sender. Twitter] <http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44mC2hjoRGQ&s=5&u=http%3a%2f%2ftwitter%2ecom%2fRiceResearch> [Image removed by sender. Flickr] <http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jLn3D4WHA&s=5&u=http%3a%2f%2fwww%2eflickr%2ecom%2fphotos%2fricephotos%2fcollections%2f> [Image removed by sender. Youtube] <http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jC2gD0TRA&s=5&u=http%3a%2f%2fwww%2eyoutube%2ecom%2fuser%2firrivideo%2ffeatured> [Image removed by sender. Scribd] <http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44mLig21AGA&s=5&u=http%3a%2f%2fwww%2escribd%2ecom%2fIRRI%5fresources> [Image removed by sender. Linkedin] <http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44j_p1GQQSQ&s=5&u=http%3a%2f%2fwww%2elinkedin%2ecom%2fcompany%2finternational-rice-research-institute> [Image removed by sender. Soundcloud] <http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jPmhmwSHw&s=5&u=https%3a%2f%2fsoundcloud%2ecom%2firri-radio> [Image removed by sender. Google+] <http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jLo1TkWGQ&s=5&u=https%3a%2f%2fplus%2egoogle%2ecom%2f103972671963502739315> The International Rice Research Institute<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jXghm0XGA&s=5&u=http%3a%2f%2firri%2eorg> is a member of the CGIAR<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44j_j1m4UGQ&s=5&u=http%3a%2f%2fwww%2ecgiar%2eorg%2f> The International Rice Research Institute<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44jXghm0XGA&s=5&u=http%3a%2f%2firri%2eorg> is a member of the CGIAR<http://scanmail.trustwave.com/?c=4062&d=otS611Q20SL86MvZAP2-57BForsTwwP44mOxgW4WSQ&s=5&u=http%3a%2f%2fcgiar%2eorg> consortium ________________________________ This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.
_______________________________________________ Owasp-modsecurity-core-rule-set mailing list Owasp-modsecurity-core-rule-set@lists.owasp.org https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
