On Wed, 23 Nov 2016, mosto...@gmail.com wrote:
My recommended architecture [1] is to have a local relay picking up logs
from each network/datacenter, add useful metadata (fromhost-ip, what
environment this is, which relay processed it, timestamp of when the log
was processed on the relay, etc) and then forward the message to a central
log system in json format [2].
This is actually what we are doing
good, this makes things much easier
On the central system, I should then have relativly clean data to deal
with. I receive it and parse the json out.
I then created a template.
t = "$timestamp $hostname $syslogtag $!msg"
note the msg is not $msg (which would be json), but rather $!msg, which is
the field inside the json that contains the original message.
I was using property(name="msg" position.from="3") to skip first JSON "{"
Another _hidden gem_
if your message is in json, you should parse it in json, not try to skip the
first characters.
set $.m = exec_template("t")
which creates a variable that contains a line like:
Nov 23 06:19:38 bifrost dhcpd: DHCPREQUEST for 10.2.0.122 from
00:90:f5:d6:7f:2a via eth2
I didn't played with exec_template, cause I didn't know if that was a "tricky
approach", recommended or anything else.
Usually whatever_exec is run as command, hence having a serious impact on
performance.
I'll have a look.
in rsyslog terms, it's not great, but as I've said elsewhere, you can do a lot
of 'not great' things and still be fast :-)
using templates is 'slow' because the template format is interpreted when you
use it. You can create a string module (sm*) that does this directly in C code,
and that makes templates extremely fast. When we first introduces them for the
standard templates, it caused a 10%+ improvement in the overall speed of
rsyslog.
prefix=%timestamp:date% %hostname:word% dhcp:
rule=dhcp,foo: DHCPREQUEST for %ip:ipv4" from %mac:word% via
%interface:word%
what's the foo part?
As someone with regex-way-of-thinking, I would've expected
rule=%prefix% DHCPREQUEST for %ip:ipv4" from %mac:word% via
%interface:word%
but it isn't.
This then parses eveything apart, and creates a variable event.tags =
["dhcp",:"foo"]
I'm lost!
a rule in liblognorm is:
rule=<tags>:<format>
most of the example rules have no tags, so you see
rule=: <something>
when you have one or more tags on a rule, the variable $!event.tags that
contains an array of the tags that are on that rule. You can have multiple tags
on one rule
so you could say that for a given log message, you want it to be processed as
multiple categories (login and ssh for example)
if $programname = "dhcp" then { call dhcprules; stop }
As with _exec, someone told me if are evil (wasn't nginx)
...and that gives you a rocket speed...
prior to rsyslog v8, if statements were significantly slower than other
conditionals. With the config optimizer in v8, all tests are equally fast.
tests still cost, so if you can replace a bank of
if <something> then /var/log/fooX
statements with a dynafile statement that used a variable that you have defined
as part of the path, you will see a huge improvement in speed.
Remember, premature optimization is the root of all evil :-)
Rsyslog is very fast and has been getting faster, start off with your config
optimized for ease of management and measure the resulting speed. If it's not
fast enough for your environment, then look at making changes. You will be
surprised at what you can do without having to worry about perfromance.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
