On Thu, Jun 6, 2013 at 1:04 PM, David Lang <[email protected]> wrote:
> On Thu, 6 Jun 2013, Xuri Nagarin wrote:
>
> I am trying to translate log flow from syslog-ng to rsyslog.
>>
>> I have a few syslog sources that collect logs from all over the
>> infrastructure and deliver it to central syslog servers that in turn
>> deliver logs to different consumers.
>>
>> At the moment, I have two consumers - one reads flat files and other
>> listens on TCP ports. For both, I need to tell them what type of event am
>> I
>> delivering to them - Linux auth, firewall event, or web log.
>>
>> Something like this:
>>
>> LogFireHose====> SyslogNG/RSyslog =========>
>> (Parse and redirect events by type)||==> If (Cisco
>> ASA) write to "FS (Cisco/ASA) & TCP DstHost: 5000"
>>
>> ||==> If (Apache Access) write to "FS (Apache/Access) & TCP DstHost: 5001"
>>
>> ||==> If (DNS logs) write to "FS (Bind/DNS) & TCP DstHost: 5002"
>>
>>
>> In Syslog-NG, every incoming message (in CEF format) is subject to a
>> parser
>> that splits the log message into eight fields. Fields 2 and 3 that are
>> vendor and product type are used to generate a template like
>> "/var/log/$vendor/$product/**logfile".
>>
>> To deliver events, by type, to a specific network destination requires
>> filters and I have 30+ different vendor/product combinations. So I end up
>> with 30+ log() statements, each with it's filter logic.
>> ----------xxxxxxxxxxxxxxx-----**-----------
>> filter f1 (if $product contains "ASA")
>> filter f2 (if $product contains "ACCESS")
>> filter f3 (if $product contains "DNS")
>> ...
>> ..
>> filter 35 (if field3 contains "blah")
>>
>> log (src=tcp;filter f1; dst=/var/log/$vendor/$product/**logfile;
>> dst=remotehost:5000)
>> log (src=tcp;filter f2; dst=/var/log/$vendor/$product/**logfile;
>> dst=remotehost:5001)
>> ...
>> ....
>> log (src=tcp;filter fx, dst=/var/log/$vendor/$product/**logfile;
>> dst=remotehost:5030)
>> ----------xxxxxxxxxxxxxxx-----**-----------
>>
>>
>> In RSyslog, I have so far written the logic to write to filesystem like
>> this:
>> --------------xxxxxxxxxxxxx---**------------
>> template(name="cefdynfile" type="string"
>> string="/var/log/joe/%msg:F,**124:2%/%msg:F,124:3%/logfile")
>>
>> ruleset(name="tcpcef") {
>> if $syslogtag=="CEF:" then { action (type="omfile" FileOwner="joe"
>> FileGroup="joe" DirOwner="joe" DirGroup="joe" DirCreateMode="0755"
>> FileCreateMode="0644" DynaFile="cefdynfile") stop }
>> }
>>
>> module(load="imtcp") # needs to be done just once
>> input(type="imtcp" port="514" ruleset="tcpcef")
>> --------------xxxxxxxxxxxxx---**------------
>>
>> Now, I am thinking how do I add rules for delivering events to tcp
>> destinations.
>>
>> I could expand the "tcpcef" ruleset and add more "if condition then
>> {action()}" statements to it, OR
>> I can write multiple rulesets, one for each filter like
>> "rule f1 { if $msg contains "blah" then action()}"
>> "rule f2 { if $msg contains "foo" then action()}"
>>
>> and then call these rules from "tcpcef" ruleset:
>>
>> ruleset(name="tcpcef") {
>> call f1
>> call f2
>> ...
>> ...
>> call fx }
>>
>> So two questions (1) Does this seem like a good way to parse/route
>> messages?
>>
>
> Well, if you are getting your messages in the CEE format, it would seem to
> me to make more sense to invoke the mmjson parsing module to extract all
> the fields. That way you aren't doing string searches, you can just do an
> if on the particular variable that you are looking for. I would expect this
> to be faster once you have a few different filters.
Sorry, logs are in CEF (Common Event Format driven by HP-ArcSight) and not
CEE :)
>
>
> (2) Which way is better for multi-threading? I read that each ruleset gets
>> it own queue and thread so I am thinking defining multiple rulesets and
>> then calling them for a master ruleset might offer better performance.
>>
>
> There's overhead to rulesets as well.
>
> Remember that premature optimization is the root of all evil. Set it up
> the simple way (single ruleset) and only look at changing it if you find
> that it isn't fast enough.
>
> using the json parser can both speed things up, but more importantly, it
> makes the rules easier to write and cleaner.
>
Yes, I am following the same approach. I am expanding the filters within a
single rule one by one and keeping an eye on performance metrics on the
host. Is there any documentation about what's better performing in terms of
string comparison within rsyslog?
Is this:
field($msg, 124, 3) == "DNS Trace Log"
better than:
field($msg, 124, 3) startswith "DNS"
??
I am expanding my ruleset this way:
----xxxxxxxxxxxxxxxxxxxxx---------
template(name="cefdynfile" type="string"
string="/var/log/joe/%msg:F,124:2%/%msg:F,124:3%/logfile")
ruleset(name="tcpcef") {
if $syslogtag=="CEF:" then { action (type="omfile" FileOwner="joe"
FileGroup="joe" DirOwner="joe" DirGroup="joe" DirCreateMode="0755"
FileCreateMode="0644" DynaFile="cefdynfile") }
if field($msg, 124, 3) == "ASA" then { action (type="omfwd"
Target="127.0.0.1" Port="5146" Protocol="tcp") stop }
if field($msg, 124, 3) startswith "DNS" then { action (type="omfwd"
Target="127.0.0.1" Port="5161" Protocol="tcp") stop }
----xxxxxxxxxxxxxxxxxxxxx---------
Thanks for your response, David.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
