On Thu, 6 Jun 2013, Xuri Nagarin wrote:
I am trying to translate log flow from syslog-ng to rsyslog.
I have a few syslog sources that collect logs from all over the
infrastructure and deliver it to central syslog servers that in turn
deliver logs to different consumers.
At the moment, I have two consumers - one reads flat files and other
listens on TCP ports. For both, I need to tell them what type of event am I
delivering to them - Linux auth, firewall event, or web log.
Something like this:
LogFireHose====> SyslogNG/RSyslog =========>
(Parse and redirect events by type)||==> If (Cisco
ASA) write to "FS (Cisco/ASA) & TCP DstHost: 5000"
||==> If (Apache Access) write to "FS (Apache/Access) & TCP DstHost: 5001"
||==> If (DNS logs) write to "FS (Bind/DNS) & TCP DstHost: 5002"
In Syslog-NG, every incoming message (in CEF format) is subject to a parser
that splits the log message into eight fields. Fields 2 and 3 that are
vendor and product type are used to generate a template like
"/var/log/$vendor/$product/logfile".
To deliver events, by type, to a specific network destination requires
filters and I have 30+ different vendor/product combinations. So I end up
with 30+ log() statements, each with it's filter logic.
----------xxxxxxxxxxxxxxx----------------
filter f1 (if $product contains "ASA")
filter f2 (if $product contains "ACCESS")
filter f3 (if $product contains "DNS")
...
..
filter 35 (if field3 contains "blah")
log (src=tcp;filter f1; dst=/var/log/$vendor/$product/logfile;
dst=remotehost:5000)
log (src=tcp;filter f2; dst=/var/log/$vendor/$product/logfile;
dst=remotehost:5001)
...
....
log (src=tcp;filter fx, dst=/var/log/$vendor/$product/logfile;
dst=remotehost:5030)
----------xxxxxxxxxxxxxxx----------------
In RSyslog, I have so far written the logic to write to filesystem like
this:
--------------xxxxxxxxxxxxx---------------
template(name="cefdynfile" type="string"
string="/var/log/joe/%msg:F,124:2%/%msg:F,124:3%/logfile")
ruleset(name="tcpcef") {
if $syslogtag=="CEF:" then { action (type="omfile" FileOwner="joe"
FileGroup="joe" DirOwner="joe" DirGroup="joe" DirCreateMode="0755"
FileCreateMode="0644" DynaFile="cefdynfile") stop }
}
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514" ruleset="tcpcef")
--------------xxxxxxxxxxxxx---------------
Now, I am thinking how do I add rules for delivering events to tcp
destinations.
I could expand the "tcpcef" ruleset and add more "if condition then
{action()}" statements to it, OR
I can write multiple rulesets, one for each filter like
"rule f1 { if $msg contains "blah" then action()}"
"rule f2 { if $msg contains "foo" then action()}"
and then call these rules from "tcpcef" ruleset:
ruleset(name="tcpcef") {
call f1
call f2
...
...
call fx }
So two questions (1) Does this seem like a good way to parse/route
messages?
Well, if you are getting your messages in the CEE format, it would seem to me to
make more sense to invoke the mmjson parsing module to extract all the fields.
That way you aren't doing string searches, you can just do an if on the
particular variable that you are looking for. I would expect this to be faster
once you have a few different filters.
(2) Which way is better for multi-threading? I read that each ruleset gets
it own queue and thread so I am thinking defining multiple rulesets and
then calling them for a master ruleset might offer better performance.
There's overhead to rulesets as well.
Remember that premature optimization is the root of all evil. Set it up the
simple way (single ruleset) and only look at changing it if you find that it
isn't fast enough.
using the json parser can both speed things up, but more importantly, it makes
the rules easier to write and cleaner.
David Lang
Thanks for patiently reading through this email even if you do not respond
:-)
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
