hi Christopher,
I don't know whether this recommendation is applicable in your environment, but
I have used Jump rules for addressing this issue.
I had a similar problem with one of my SEC instances which is processing a
massive event stream with hundreds of rules. Although I separated my rules into
several config files, I still had a problem that every event was matched
against all rules which limited event throughput to 1500-2000 events per
second.
However, after introducing Jump rules in SEC 2.5.0, I created one rule file
called "master" which is used for matching *all* input. The master file
contains Jump rules with simple string/regexp patterns which direct events to
relevant "slave" rule files for final processing. For example, you could have
the following Jump rule:
type=Jump
ptype=RegExp
pattern=sshd\[\d+\]:
cfset=sshd-rules
which will direct all events containing a string sshd[<processid>]: to
configuration file set called "sshd-rules". If you have a separate rule file
with the following Options rule:
type=Options
joincfset=sshd-rules
procallin=no
the rest of the rules in this file will be used from processing events from the
above Jump rule (since 'procallin' option is set to 'no', the rules will not be
used for matching all events from input files, but from Jump rules only).
After implementing this simple approach for my SEC instance, the event
throughput increased by the factor of 3. Note that you are not limited for
having two configuration file layers, but you could build a tree with several
intermediate layers if you like.
Hope this helps,
risto
--- On Tue, 10/12/10, Morris, Christopher
<christopher.mor...@sabre-holdings.com> wrote:
From: Morris, Christopher <christopher.mor...@sabre-holdings.com>
Subject: [Simple-evcorr-users] Best practice question
To: "simple-evcorr-users@lists.sourceforge.net"
<simple-evcorr-users@lists.sourceforge.net>
Date: Tuesday, October 12, 2010, 8:12 PM
I use SEC to monitor UNIX syslog messages on a central
loghost server.
When I started using SEC a single config file was sufficient.
But as time went on and number of rules grew it became clear that my single
config file was a maintenance problem.
To make matters worse, I use a philosophy whereby I create a
SEC rule for every known message/issue -- and then any unknown message “falls
through” all the rules and is reported as an unknown event so that I can
examine it and of course make a new rule. (The reasoning behind this
process is that I don’t necessarily know what kinds of events I am
looking for and I need to be aware of new or unknown events. Our
environment is highly dynamic and so have to be able to adapt to changing
conditions.)
As you can guess -- besides a maintenance problem, there is
also a performance problem as each message may be examined by lots of
rules. I’ve mitigated this problem to some degree by creating
sections in our configuration file... There is a section at the top
called ‘high-volume’ that intercepts and handles events that are
known to be voluminous in nature. And then there’s a ‘temporary’
section to deal with short term or unusual conditions. Finally, the
remaining rules are ordered by the syslog ‘tag’. I sometimes
use SIGUSR1 to dump the rule-usage-frequency and may will reorder the rules to
reduce the load on the system and to improve message processing. But of
course this adds to the maintenance burden... (BTW, our loghost is a Sun
R240 system.)
I’ve experimented with creating rules in separate
config files - but this has two problems from my perspective: One is that
it doesn’t help the problem of unnecessary examination of events by rules
that don’t really apply to that ruleset -- and (2) is that my
‘philosophy/strategy’
of using a set of rules to filter out known events and reporting exceptions
breaks down when I use multiple config files... Or does it?
I guess I’m asking if there are some ideas/suggestions
for rules/patters/strategies that I might be able to adopt to make our SEC
configuration better and to make management & maintenance easier given the
style and goals we have.
Thanks!
-----Inline Attachment Follows-----
------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb
-----Inline Attachment Follows-----
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
