On Wed, 23 Nov 2016, [email protected] wrote:
The ugly way to do this would be a series of
if $programname = "group/appX" then set $.owner =
"\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",";
do having multiple templates affect performance? (what I really noticed it's
they affect loading time!)
not really, but we haven't had anyone experiment with thousands of them, so it's
possible, but unlikely that there would be a measureable slowdown as rsyslog
finds the right one to use.
The bigger overhead is in interpreting the template, that's where simplifying it
to be $! or $!foo would be a big win (or writing a string module)
a far more elegant way to do this would be to do a table lookup on the
programname and have it return the string.
I have readed about how lookup tables can be used for geoIP. Could you
provide a link to doc where there's an example?
there isn't a good writeup, but if you read on how to use the maxmind database,
the perl example has you create an array where the first element is the decimal
equivalent of the first IP address that matches the data.
This is exactly the structure that a sparse array lookup table is intended for.
I beleive there is a function that will take an IPv4 address and return a
decimal number (if not, we need to add one). Use that function to create a
number, lookup the number in the lookup table, and have it return the data.
you can also simplify the template a bit. Instead of crafting the json in
the template, create a variable that has what you want in/under it and
output that variable. but compared to collapsing all the templates
together, that's a minor change :-)
one variable for each file and one template which use it, isnt it?
$!foo!bar = "abc" maps to {"foo": { "bar": "abc" } } in json and if you put
%$!foo% in a template, what you will get is '{ "bar": "abc" }'
so where you had group, unit, app, msg, and a couple other things, and then
combined them with {}," into a json string, you could instead do
set $!foo!group = "A";
set $!foo!unit = "b";
set $!foo!msg = $!msg;
...
and then replace all that hard-to-read json construction in the template with
$!foo
I question the value of having a separate sending queue for each app. I
think it's better to send them in one combined firehose and split them on
the receiving side. It makes it less disruptive when you find you want to
change the groupings of things and all those queues on the sender can eat
up a lot of ram.
Probably this is because i came from redis.
That's what I'm thinking. With logstash you are forced to use something external
for queueing and lots of separate instances (and separate parser sets) or things
just don't work well.
With rsyslog, the performance is 100-1000x as fast, and a lot of the stuff is
built-in, so you don't need to split things up as much, and the reduction in the
communications overhead adds to your wins.
Talking about elastic, probably ingest node would be the best option, while
having index name as metadata.
There are two approaches, and I haven't tried them under fire on a ES cluster to
know which is the best.
dedicate a node to ingest the data
spread the traffic across many different nodes and have a local copy of rsyslog
receive the data and push it into the local ES instance.
I suspect that properly managed, a dedicated injest node will be a win.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
