On Tue, 22 Oct 2013, David Lang wrote:
If you are doing a lot of complicated regex stuff for creating strings,
strongly consider writing (or asking Adiscon to write) a custom string
generation module. It's very likely that a bit of C code can do things far
more efficently than what you have to do in the template configuation to
create the paths.
I'm strongly in favor of pushing message cleanups out to the first-tier relay
systems, you probably have more of them, and as such you can allocate more
CPU to cleanup work.
when you use JSON in your relay layer you can add additional metadata to the
log message without confusing the final destination (things like the real
source IP of the log message, is this dev/QA/prod/DR, what business unit is
this for so alerts can go to the right people, etc)
re: TCP vs UDP, if you are just going over a local switch, UDP is very
reliable, the more potential chokepoints in the path, the more valuable TCP
becomes. RELP really becomes needed when the path becomes long and there are
a lot of messages in flight, or you end up with connections with a relatively
high probability of silently failing (WAN links, or firewalls that can
timeout connections or failover/restart and loose track of existing
connections are great examples)
by the way, metadata could be application name or other extracted information
that you then use directly in your templates on the central servers without
having to parse it out.
David Lang
David Lang
On Tue, 22 Oct 2013, Boylan, James wrote:
The performance problems are the expected ones. Our current environment
layout has two large log archiving servers using rather complex regex for
generating the dynafile name. Added to that, we are looking at adding
output to elasticsearch, but to do so we need to use an even more complex
set of regex to build the JSON output. So you can imagine the negative
impact to the environment when handling that many messages per second.
I've just finished rewriting the config in the new format and with the
patch for re_extract that Rainer sent I've implemented all of the regex in
local variables since there was a lot of duplicate regex comparisons
happening. That's helped a lot.
I'm looking at breaking it out so the relay layer only acts as traffic
manager and rebuilding the Rsyslog message to force a FQDN into the server
name field. Then it will be passed to the specific server pools for
handling archiving, elasticsearch and hadoop.
All of this is done with TCP as we needed to make sure that packets reached
the destination as reliably as possible without adding too much overhead.
(I'm still considering relp on the transmission portion between the relay
layer and the back end services, but I haven't made a decision yet.)
-- James
----- Reply message -----
From: "David Lang" <[email protected]>
To: "rsyslog-users" <[email protected]>
Subject: [rsyslog] Large Scale Rsyslog deployment
Date: Tue, Oct 22, 2013 5:03 am
On Tue, 22 Oct 2013, Boylan, James wrote:
I know there are several individuals on the list that manage a large scale
Rsyslog environment handle 70k to 100k+ messages per second.
I was wondering if they could share roughly the number of Rsyslog
instances
running on their relay layer. I'm hoping to get confirmation on the
numbers
I'm looking at. Commenting on if they are using UDP versus TCP would be
helpful as well.
I am vastly overprovisioned on the relay layer as I put in relays per
environment. I use UDP from the application servers to the relay and then
UDP if
it's relaying on a local subnet (multi-homed relay systems in one
datacenter) or
RELP if it's relaying to a remote network (especially if it goes over a
WAN)
in my older, all-local setup I put 6 pairs of relays in one datacenter, in
the
newer, larger environment I started with 3 pairs of relays per datacenter,
but
expect that more will be needed eventually (I am no longer at the company,
so I
won't be building out that system further). I have all of these relay to
one
pair of core relay boxes that then distribute the logs to the different
analysis
boxes.
It depends a lot on what you are doing on the relay boxes. If you are just
relaying messages without modifying them, you should be able to get up to
gig-E
wire speed on a single box (although we recently found a bottleneck that
triggered imudp to be able to be multithreaded to handle that load)
if you are modifying the messages, doing encryption, etc you may find that
you
run into performance limitations sooner.
If you are running into performance problems, we'd be interested in hearing
details and trying to address them.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
