In message <CAJ69-dFBdhn7DyqSYYaO9QVHpdw0u1_Nsk3=x0EBc8=ysd7...@mail.gmail.com> , =?UTF-8?Q?Richard_Ostrochovsk=C3=BD?= writes: >Hello friends, > >I have SEC monitoring over 50 log files with various correlations, and it >is consuming 100% of single CPU (luckily on 10-CPU machine, so not whole >system affected, as SEC is single-CPU application). > >This could mean, that SEC does not prosecute processing of all rules, and I >am curious, what are possible effects, if this means increasing delays >(first in, processing, first out), or skipping some lines from input files, >or anything other (?). > >And how to troubleshoot, finding bottlenecks. I can see quantities of log >messages per contexts or log files in sec.dump, this is some indicator. Are >there also other indicators? Is it possible, somehow, see also processing >times of patterns (per rules)?
I have found two operations that are large users of cpu in SEC. 1) If you have a lot of contexts, context lookups can eat cpu. I solved a problem where sec used 50% of a cpu by actively deleting contexts as soon as I could rather than lettng them time out. You can use the USR1 signal (IIRC see manpage) to get a dump of all contexts. I don't remember how many I had but it was a lot. Once I started actively deleting contexts the cpu dropped to 1-2% with ~100-500 events per second though the system. 2) Inefficient regexps that require a lot of backtracking can kill your cpu. I don't know how you would find these. You don't say how your rulesets are constructed, but if you have say 100 rules and the most often hit rules are at position 99 and 100, you are going to waste a lot of time applying regexps (even if they are efficient). Again a dump file will tell you which rules are being hit. Move the most used rules so they are applied earlier in you rulesets. Also you can create rules so they are applied in a tree like structure. Look at the manpage for rulesets. This structure allows you to have a layer of filter rules (e.g. look for ssh in the log line) and only apply rulesets for ssh to the log event. This can cut down dramaticaly on the number of regexps that have to be applied and discarded. Also you can parse the event in one rule and then use more efficient means to trigger rules (matching on a subset of the original event). I mentiond as an offhand remark to Risto a profile mode that would count not only every rule that lead to an action, but every time the rule executed its regular expression. Having some sort of profile mode (not to be run in production) would help identify these sorts of issues. Good luck and when you figure it out let us know how you found and fixed it. -- -- rouilj John Rouillard =========================================================================== My employers don't acknowledge my existence much less my opinions. _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users