I have used Rsyslog in the past to run a 66 node array (6 core VMs) to process 6M log-events / second (avg size 1.5 KB), which boils down to 69 Gbps of log traffic. This was being received over ptcp and delivered to the backend over omkafka.
Rsyslog and HDFS (of Hadoop) were by-far the most trouble-free components in this architecture. In addition to data-plane, Rsyslog did accounting based on custom fields (such as cluster-id or application-id), reported those stats, performed L7 service defence (blocking abuse) and helped get data out for escallation to L3 defense and much more. We saw everything break at this scale, including Rsyslog, but had to make very few and relatively tiny changes to Rsyslog (all merged upstream, btw) to get things going whereas most other components had to be replaced or large parts of them had to be re-written. So when it comes to stability and performance, I totally swear by Rsyslog. When building such large systems one doesn't want to take over maintainership of components used (which is sometimes the sorry state things get into when upstream doesn't acknowledge the problems or doesn't accept fixes for it). We were at 0 locally patched changes on Rsyslog build, thanks to Rsyslog community which is super responsive and helpful. If I ever build such a system again, I would choose Rsyslog for log-mux with my eyes closed. On Wed, Feb 6, 2019 at 5:30 AM Dave Caplinger via rsyslog <rsyslog@lists.adiscon.com> wrote: > > Hi Vishal, > > It's been many years since we switched from (open source) syslog-ng to > rsyslog. We did it because we were struggling with configuration complexity > and performance issues, and also because Balabit supported certain features > (such as local disk buffering) only in the commercial-only version, but their > pricing model was not justifiable for us at the time. I'm sure that > syslog-ng has changed quite a bit in that time so a comparison today may not > turn out the same way, but we've been very happy with rsyslog. If syslog > collection is important to you or your organization, my opinion is that > rsyslog is the best choice. The prompt adoption of new technologies over the > years such as json, liblognorm, stream-compression, elasticsearch, kafka, and > many others demonstrate that this project continues to have an active user > and development community. > > As for performance: we are certainly not the largest-volume users of rsyslog, > but as one anecdotal example I happen to have handy, we've used rsyslog to > collect over 24 million logs (around 12.5 TB) per day. You will need to > learn about action queues, buffer sizing, pstats, and tuning, but rsyslog can > handle it. > > Additionally, we've been happy to help financially support the rsyslog > project by maintaining an Adiscon support contract, as several others on this > mailing-list do as well (just to clarify that unwillingness to pay was not > the reason we decided not to go with syslog-ng pro). > > Best regards, > > -- > Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security > Corporation > > > On Feb 4, 2019, at 12:45 AM, vishal via rsyslog <rsyslog@lists.adiscon.com> > > wrote: > > > > Hi, > > I am evaluating rsyslog and syslogng for our project. > > Though aware of some of the differences and pros and cons, but still would > > like to know the differences which users have faced and evaluated in terms > > of ease of use, robustness, handling huge volumes of logs and deployment > > scenarios (single host to multi host cluster) , and if there are any other > > important areas to be considered. > > > > The general deployment would be, > > > > Log sources -> rsyslog/syslogng -> elasticsearch > > > > > > Thanks. > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. > > > Confidentiality Notice: The content of this communication, along with any > attachments, is covered by federal and state law governing electronic > communications and may contain confidential and legally privileged > information. If the reader of this message is not the intended recipient, you > are hereby notified that any dissemination, distribution, use or copying of > the information contained herein is strictly prohibited. If you have received > this communication in error, please immediately contact us by telephone at > 402.361.3000 or e-mail security-ameri...@nttsecurity.com. > > Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT > Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
