hi Richard,
> >> You mentioned that existing HPOM configuration was converted on 1:1 basis >> -- does that mean that SEC is configured to use a number of rule files, >> where each rule file corresponds to some HPOM policy, and all rules are >> applied against all input messages from 50 input files? >> > > 50 HPOM policies (of type LOGFILE) = 50 SEC rules files > > This I meant to be single level of hierarchy, applying conditions only on > log files they should be applied. > > SEC is starting with -input=logfilepath=contextname for each of them > (which should mean also implicit enablement of -intcontext, as it is not > enabled explicitly). > > There is next level of hierarchy in each rules file, implemented by using > context conditions (as each log file has not only its "default" context, as > navigated from -input command option, but also other "derived" contexts, > present only in that particular rules file... names of contexts beginning > with the one used also on commandline from -input), and patterns routing > events into another GoTo sections for further "parametrized deduplications". > > There are hundreds of rules in total in all rules files. > > Since such use of rules is not very efficient, perhaps you could introduce >> hierarchical rule arrangement in the following way. >> > > For now, I am not sure, if what I described above, is negative or positive > according to your assumption. > As I can see from above description, each rule file has been optimized with GoTo statements, so that irrelevant rules for any given event can be skipped. Such optimizations are always very useful and can be used *together* with hierarchical rule arrangement, yielding even greater performance benefits (I have often combined both techniques in larger rule sets, in order to make them more scalable). > First, assign each rule file to a configuration file set with Options >> rule. For example, if you have an HPOM policy called "sshd", you can use >> the following Options rule in the beginning of corresponding rule file: >> >> type=Options >> joincfset=sshd >> procallin=no >> >> Secondly, run SEC with --intcontexts flag which will set an internal >> context for each file (e.g., the file /var/log/secure will have the context >> _FILE_EVENT_/var/log/secure by default, but you can also set a custom >> context name in command line). Finally, create a special rule file (e.g., >> main.sec) which will route messages to relevant rule files. >> > > In our setup, there is no "main.sec" and we rely on > -input=logfilename=contextname for each input log file, and then context > condtions in all rules, so patterns are being processed only on events from > log file they belong to. It seems to me, that this is not as optimal as > your "main.sec" jumping suggestion, as I understand, that context > conditions are still being evaluated also for events they does not belong > to, and maybe it could be further optimized (while preserving our concept) > by adding a rule on the top of each rules file, continuing with DontCont > with anything, what does not belong to contexts with names "beginning with" > contextname (as we have practise each context name in rules file having the > same prefix). > Did you mean adding a rule like this in top of each rule file: type=Suppress ptype=TValue pattern=TRUE context=[ ! _FILE_EVENT_/var/log/secure ] <here come the rules which are relevant to /var/log/secure> ... If so, this approach would also work for reducing the CPU load, and is definitely much better that having all rules being matched against all events. However, this approach means that all 50 rules in top of 50 rule files would still be evaluated against all events. On the other hand, if you have one main.sec file with 50 Jump rules, you can move most frequently matching Jump rules to the top of main.sec, having a situation where for many events only few Jump rules would have to be evaluated before match is found. Also, since you have 50 input files, perhaps you can also use another optimization for polling input files. If most input files are infrequently modified, you can use --check-timeout option for increasing time interval between polling files with no new data. By default, SEC will sleep for 0.1 seconds if no new data have been read from input files, but if you use option like --check-timeout=3, file that had no new data during last poll will not be checked for the following 3 seconds. Using --check-timeout option can also reduce the CPU load of your systems if most input files are rarely updated (say, once per 10-20 minutes). > But I will wait for your potential feedback yet, and then I will consider > suggestions of further optimizations in our team. > > Thank you. > > Richard > > Hopefully you will find this feedback is useful. kind regards, risto
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
