The performance problems are the expected ones. Our current environment layout has two large log archiving servers using rather complex regex for generating the dynafile name. Added to that, we are looking at adding output to elasticsearch, but to do so we need to use an even more complex set of regex to build the JSON output. So you can imagine the negative impact to the environment when handling that many messages per second.
I've just finished rewriting the config in the new format and with the patch for re_extract that Rainer sent I've implemented all of the regex in local variables since there was a lot of duplicate regex comparisons happening. That's helped a lot. I'm looking at breaking it out so the relay layer only acts as traffic manager and rebuilding the Rsyslog message to force a FQDN into the server name field. Then it will be passed to the specific server pools for handling archiving, elasticsearch and hadoop. All of this is done with TCP as we needed to make sure that packets reached the destination as reliably as possible without adding too much overhead. (I'm still considering relp on the transmission portion between the relay layer and the back end services, but I haven't made a decision yet.) -- James ----- Reply message ----- From: "David Lang" <[email protected]> To: "rsyslog-users" <[email protected]> Subject: [rsyslog] Large Scale Rsyslog deployment Date: Tue, Oct 22, 2013 5:03 am On Tue, 22 Oct 2013, Boylan, James wrote: > I know there are several individuals on the list that manage a large scale > Rsyslog environment handle 70k to 100k+ messages per second. > > I was wondering if they could share roughly the number of Rsyslog instances > running on their relay layer. I'm hoping to get confirmation on the numbers > I'm looking at. Commenting on if they are using UDP versus TCP would be > helpful as well. I am vastly overprovisioned on the relay layer as I put in relays per environment. I use UDP from the application servers to the relay and then UDP if it's relaying on a local subnet (multi-homed relay systems in one datacenter) or RELP if it's relaying to a remote network (especially if it goes over a WAN) in my older, all-local setup I put 6 pairs of relays in one datacenter, in the newer, larger environment I started with 3 pairs of relays per datacenter, but expect that more will be needed eventually (I am no longer at the company, so I won't be building out that system further). I have all of these relay to one pair of core relay boxes that then distribute the logs to the different analysis boxes. It depends a lot on what you are doing on the relay boxes. If you are just relaying messages without modifying them, you should be able to get up to gig-E wire speed on a single box (although we recently found a bottleneck that triggered imudp to be able to be multithreaded to handle that load) if you are modifying the messages, doing encryption, etc you may find that you run into performance limitations sooner. If you are running into performance problems, we'd be interested in hearing details and trying to address them. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
