hi Richard,
> We were doing log monitoring migration from HPOM to open-source monitoring > tool, and using SEC for duplicate events flow reduction before passing to > monitoring agent, in the manner as HPOM agent with built-in correlations > was used, so the design of rules and correlations is tributary to how it > was implemented in HPOM. There were hundreds to thousands of pattern > conditions in HPOM per host, and the structure of their sections was as > follows: > > - HPOM: suppress unmatched conditions -> SEC: Suppress with NRegExp > - HPOM: suppress matched conditions -> SEC: Suppress with RegExp > - HPOM: message conditions (with configured time-based correlations) > -> SEC: Single with RegExp and GoTo -> duplicate suppress time-based > correlations, each consisting of 3-4 subsequent rules (Single, > PairWithWindow, SingleWithSuppress, depending on duplicate suppress > correlation type) > > We decided to automate conversion of HPOM configurations to SEC rules, so > here was not too much space for conceptual improvements over HPOM concepts > (e.g. by doing deeper analysis of configurations and actual log traffic), > and we relied on the premise, that those HPOM configurations are OK, and > tuned by years of development and operations, so the automated conversion > was 1:1. > > Cca 50 log files per host are of several types (according to message > structure), but each file was monitored in HPOM independently on each > other, therefore after 1:1 conversion also in SEC is each file monitoring > independently, however, there is some maybe uglier "configuration > redundancy" for log files of the same type, as it was in HPOM. The static > order of conditions in HPOM is preserved also in generated SEC rules. > > Since I have used HPOM in the past, perhaps I can offer some comments and advise here. As far as I remember, HPOM agent does not support hierarchical arrangement of log file monitoring policies and rules. You mentioned that existing HPOM configuration was converted on 1:1 basis -- does that mean that SEC is configured to use a number of rule files, where each rule file corresponds to some HPOM policy, and all rules are applied against all input messages from 50 input files? Since such use of rules is not very efficient, perhaps you could introduce hierarchical rule arrangement in the following way. First, assign each rule file to a configuration file set with Options rule. For example, if you have an HPOM policy called "sshd", you can use the following Options rule in the beginning of corresponding rule file: type=Options joincfset=sshd procallin=no Secondly, run SEC with --intcontexts flag which will set an internal context for each file (e.g., the file /var/log/secure will have the context _FILE_EVENT_/var/log/secure by default, but you can also set a custom context name in command line). Finally, create a special rule file (e.g., main.sec) which will route messages to relevant rule files. For example, suppose that HPOM policies "sshd" and "sudo" have been used for /var/log/secure, and there are two SEC rule files that contain Options rules as described above. For handling messages from /var/log/secure, you can enter the following Jump rule into main.sec rule file: type=Jump context=[ _FILE_EVENT_/var/log/secure ] ptype=TValue pattern=True desc=direct /var/log/secure events to relevant rules files cfset=sshd sudo Such configuration ensures that messages of /var/log/secure are being matched against relevant rules only, and rule files not associated with /var/log/secure will not be applied. Also, only the Jump rules from main.sec are applied against the entire event stream, while rules from other files receive their input from Jump rules in main.sec and are thus applied only for relevant messages. Perhaps you have already applied this technique, but if not, I would definitely recommend to use this optimization, since it is very likely to reduce the CPU load. hope this helps, risto
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
