If you are doing a lot of complicated regex stuff for creating strings, strongly
consider writing (or asking Adiscon to write) a custom string generation module.
It's very likely that a bit of C code can do things far more efficently than
what you have to do in the template configuation to create the paths.
I'm strongly in favor of pushing message cleanups out to the first-tier relay
systems, you probably have more of them, and as such you can allocate more CPU
to cleanup work.
when you use JSON in your relay layer you can add additional metadata to the log
message without confusing the final destination (things like the real source IP
of the log message, is this dev/QA/prod/DR, what business unit is this for so
alerts can go to the right people, etc)
re: TCP vs UDP, if you are just going over a local switch, UDP is very reliable,
the more potential chokepoints in the path, the more valuable TCP becomes. RELP
really becomes needed when the path becomes long and there are a lot of messages
in flight, or you end up with connections with a relatively high probability of
silently failing (WAN links, or firewalls that can timeout connections or
failover/restart and loose track of existing connections are great examples)
David Lang
On Tue, 22 Oct 2013, Boylan, James wrote:
The performance problems are the expected ones. Our current environment layout
has two large log archiving servers using rather complex regex for generating
the dynafile name. Added to that, we are looking at adding output to
elasticsearch, but to do so we need to use an even more complex set of regex
to build the JSON output. So you can imagine the negative impact to the
environment when handling that many messages per second.
I've just finished rewriting the config in the new format and with the patch
for re_extract that Rainer sent I've implemented all of the regex in local
variables since there was a lot of duplicate regex comparisons happening.
That's helped a lot.
I'm looking at breaking it out so the relay layer only acts as traffic manager
and rebuilding the Rsyslog message to force a FQDN into the server name field.
Then it will be passed to the specific server pools for handling archiving,
elasticsearch and hadoop.
All of this is done with TCP as we needed to make sure that packets reached
the destination as reliably as possible without adding too much overhead. (I'm
still considering relp on the transmission portion between the relay layer and
the back end services, but I haven't made a decision yet.)
-- James
----- Reply message -----
From: "David Lang" <[email protected]>
To: "rsyslog-users" <[email protected]>
Subject: [rsyslog] Large Scale Rsyslog deployment
Date: Tue, Oct 22, 2013 5:03 am
On Tue, 22 Oct 2013, Boylan, James wrote:
I know there are several individuals on the list that manage a large scale
Rsyslog environment handle 70k to 100k+ messages per second.
I was wondering if they could share roughly the number of Rsyslog instances
running on their relay layer. I'm hoping to get confirmation on the numbers
I'm looking at. Commenting on if they are using UDP versus TCP would be
helpful as well.
I am vastly overprovisioned on the relay layer as I put in relays per
environment. I use UDP from the application servers to the relay and then UDP if
it's relaying on a local subnet (multi-homed relay systems in one datacenter) or
RELP if it's relaying to a remote network (especially if it goes over a WAN)
in my older, all-local setup I put 6 pairs of relays in one datacenter, in the
newer, larger environment I started with 3 pairs of relays per datacenter, but
expect that more will be needed eventually (I am no longer at the company, so I
won't be building out that system further). I have all of these relay to one
pair of core relay boxes that then distribute the logs to the different analysis
boxes.
It depends a lot on what you are doing on the relay boxes. If you are just
relaying messages without modifying them, you should be able to get up to gig-E
wire speed on a single box (although we recently found a bottleneck that
triggered imudp to be able to be multithreaded to handle that load)
if you are modifying the messages, doing encryption, etc you may find that you
run into performance limitations sooner.
If you are running into performance problems, we'd be interested in hearing
details and trying to address them.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
