Ryan I have implemented the experimental rules on a couple of servers, one in a production environment. Having read the blog post at:
http://blog.spiderlabs.com/2011/02/modsecurity-advanced-topic-of-the-week-real-time-application-profiling.html and your reply to Lucas in this thread, I have some additional comments. None of these are meant to criticise the excellent work so far, and may well reflect ideas the team already has, or are just restating concepts in related work from Ivan Ristic, Ofer Shezaf and Christian Bockermann. Some of these are of course just a dream wish list. Anyway here goes... Configuration --------------- C1: I had to create an empty "modsecurity_40_profiler_ignore.data" for the rules to work "as is" i.e. this file is not in the CRS 2.1.2 distribution. Is the syntax of this file one URL per line? C2: Apart from ignoring certain URLs, could we specify some parameters to ignore (on some URLs) and on some HTTP methods? i.e. ignore params if GET, but enforce if POST. C3: Could the thresholds (50 and 100) be set near the top of the rule file, instead of within the SecAction of Step 1? C4: Thresholds per host/site? C5: Some application owners may know all the allowable entry points.... it would be fantastic to have a way of defining that profile (in XML?) which the CRS profiler could i) use as a base starting point, or ii) use as a mandatory profile, or iii) use to compare against when reporting discrepancies. This would also help to group similar URLs (e.g. paths when URL rewriting is in use). C6: Ability to modify model in C5 in real time (e.g. as URLs are added created when writing a new blog post, or pages are added in a CMS) C7: Ability for model in C5 to have the idea of "strict" and "allowable" parameter value definitions. Example 1: we might want a phone number to be all numbers and spaces, but let's not get too upset with hyphens and parentheses. Example 2: A trailing line break on a password could just be from copy & paste, not an attack. So the profiler would then need to report whether a parameter value just met the allowable format, or failed that too. Method ---------- M1: Accessing the same URL on multiple virtual hosts, using the same instance of ModSecurity (and with the same web browser) seems to create an aggregated profile rather than a profile per host (but I may be wrong): http://www.example1.com/profiler.txt?site=2 http://www.example2com/profiler.txt?site=1&day=wednesday Is there a way to profile each host separately? M2: Could (or Is?) the HTTP method be part of the profile so there could be different parameters for GET and POST for example. M3: Further data types (bit, SQL types, UUID, lists of integers, etc), and perhaps user-definable data types - both of these would be of more use if C5 is implemented. M4: Ability to allow null values for parameters e.g. normally 0 or 1 but could be NULL M5: Build result into anomaly scoring. Data Storage ------------------- D1: Do you have any references for the format of this file, and of the profiler key records, so we can try to extract data and convert it into other formats? Syndication of result ------------------------------ S1: Could the "result" be added as another X-WAF-... header S2: This one isn't Profiling-specific, but currently the X-WAF- headers are added to the request... could there be a method to pass data on by URL (e.g. like CSP reporting mechanism), so that say even if something is blocked it is possible to inform other devices of the event? The data sent could be configurable, but might include what's currently in the X-WAF- headers, plus the original header content. Colin On 18 February 2011 15:11, Ryan Barnett <[email protected]> wrote: > > On 2/18/11 9:52 AM, "Lucas Ferreira" <[email protected]> wrote: > >>Hello Ryan, >> >>after reading your blog post, I still have some doubts: >> >>How does the profiling work with regard to restarting the web server? >>Are the collections saved to disk? > > Yes - the data is saved to persistent storage files on disk. This would > keep the data across Apache restarts. > >> >>How can the administrator restart the profiling process? Is it >>possible to selectively restart profiling based on host, URL or other >>application identifier? > > Great question. We actually were talking a bit about some options for > profile management. One option that we are thinking of would be to use > dynamic updates based on access requests from "trusted" clients. Take a > look at the previous blog post on GeoIP data where we talk about dynamic > updates - > http://blog.spiderlabs.com/2010/10/detecting-malice-with-modsecurity-geoloc > ation-data.html > > The idea would be that if you wanted to remove/relearn a profile for a > URL, you could have an authorized client send a specific request to that > URL. Based on the source IP, UA + Parameter (maybe > ?relearn_profile=true), then the saved resource collection data for that > URL would be removed and the relearning would commence. There are some > other ideas that we we need to test which could possibly include tracking > an increase in alerts across multiple clients as this might indicate an > application change and could initiate an auto-relearn. > > This is something that we are planning to add in future versions. > > Please keep the comments coming! We want people to test so that we can > improve the capabilities. > > Cheers, > Ryan > >> >>Thanks, >> >>Lucas >> >>On Thu, Feb 17, 2011 at 16:27, Ryan Barnett <[email protected]> >>wrote: >>> Hello everyone, >>> This email will most likely come as very welcomed news to most of you. >>>We have just released a ruleset to the OWASP CRS that implements a basic >>>framework for real-time application profiling - >>> >>>http://blog.spiderlabs.com/2011/02/modsecurity-advanced-topic-of-the-week >>>-real-time-application-profiling.html >>> >>> This initial version of the rules has the ability to profile and >>>enforce the following on a per-resource basis: >>> >>> * Request Method(s) >>> * Number of Parameters >>> * Parameter Names >>> * Parameter Length Ranges >>> * Parameter Types - numeric or alpha >>> >>> Please test out this ruleset and provide feedback on the OWASP CRS >>>mail-list. >>> >>> Cheers, >>> Ryan >>> >>> >>> ________________________________ >>> This transmission may contain information that is privileged, >>>confidential, and/or exempt from disclosure under applicable law. If you >>>are not the intended recipient, you are hereby notified that any >>>disclosure, copying, distribution, or use of the information contained >>>herein (including any reliance thereon) is STRICTLY PROHIBITED. If you >>>received this transmission in error, please immediately contact the >>>sender and destroy the material in its entirety, whether in electronic >>>or hard copy format. >>> >>> _______________________________________________ >>> Owasp-modsecurity-core-rule-set mailing list >>> [email protected] >>> https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set >>> >> >> >> >>-- >>Homo sapiens non urinat in ventum. >> > > > _______________________________________________ > Owasp-modsecurity-core-rule-set mailing list > [email protected] > https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set > _______________________________________________ Owasp-modsecurity-core-rule-set mailing list [email protected] https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
