I using similar approach as David mention. I processing syslog messages from network devices (various Vendors like Arista, Cisco, Juniper etc.).
We have about 7k patterns defined for match. As SEC cannot handle such regexp volume, I using syslog-ng PatternDB for pattern matching, classification and tagging. Than all matched logs are written in normalized format into file what monitored by SEC and pre-defined correlation schema is applied. This help me to reduce flow into SEC to 30 msg/s while incoming syslog message flow is about ~20k msg/s (from about 20k unique network devices). My SEC instance is eating ~ 20 – 30 % of CPU (with about 500 active event correlation operations with ~ 4k active contexts in avg). ________________________________ Od: David Lang <da...@lang.hm> Odoslané: štvrtok 26. marca 2020 1:24 Komu: Richard Ostrochovský <richard.ostrochov...@gmail.com> Kópia: simple-evcorr-users@lists.sourceforge.net <simple-evcorr-users@lists.sourceforge.net> Predmet: Re: [Simple-evcorr-users] SEC CPU utilization Much of the time you can trivially split your rules and then run multiple copies of SEC, each processing a subnet of the rules on the logs that will match them. Rsyslog has a very efficient parsing capability (mmnormalize), you can use it to just classify the logs, or you can go a step further and have it extract the relevant fields and pass them to SEC in ways that make it cheaper to parse and process. where you do have rules that correlate across different types of logs, you may be able to just combine those logs into one ruleset, or you can have parallel instances output messages to say that their part of the correlation has tested true and then have another instance that processes these partial correlation messages and decide if the combined correlation is matched. David Lang _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
