I'm in the process of protecting my loghosts running syslog-ng. syslog-ng injects the msgs into mysql. The msgs are parsed into their appropriate fields prior to db injection. I can maintain a steady throughput of 3k msgs/sec.
I ran into a problem a fews week back. A syslogging host went off the rails and pushed more msgs than I could process. Msgs spewed in, I couldn't drain the queue fast enough, mysql consumed more memory than we should've allotted, fire fell from the sky. I've got snort/snortsam in place as part of a brute force syslog DOS protection mechanism. If msgs come to the door, faster than can be handled, a BLOCK is put into place. I wanted to compress the msgs in order to eliminate writing 10k duplicate msgs...or the useless "last message repeated n times", in the hope of writing a single message with start/stop times and # of msgs seen in that timeframe. I believe I have sufficient cpu and memory(X6250) with a 10Gb feed, unfortunately the long pole are SATA 500GB 8M cache drives. The iptable BLOCK is a last resort to protect the collector from the forces of evil. tnx --joe On Oct 22, 2009, at 16:15:07:EDT, da...@lang.hm wrote: > On Thu, 22 Oct 2009, John P. Rouillard wrote: > >> In message <alpine.deb.2.00.0910221240160.11...@asgard.lang.hm>, >> da...@lang.hm writes: >>> On Thu, 22 Oct 2009, Clayton Dukes wrote: >>>> If you're using the latest version of php-syslog-ng it has a >>>> built in >>>> deduplication function. >>>> http://code.google.com/p/php-syslog-ng >>> >>> most syslog daemons have the ability to say 'last message repeated X >>> times' >>> >>> rsyslog has an option to include the message that was repeated at >>> the end >>> of the 'last message repeated' line >> >> Yeah I usually disable the "last message repeated X times" first >> thing >> 8-). However if you are getting multiple events/second then it's >> usually easy enough to work around. >> >>> I have rsyslog handling tens of thousands of messages/sec >>> currently with >>> fairly low cpu utilization (<20% of one core) and tested it about 6 >>> months ago up to ~80K messages/sec sustained, >300K messages/sec >>> peak >>> (essentially wire speed of Gig-E). the current dev version has >>> significant improvements that should allow this to go significantly >>> higher, but the kinks are still being worked out of it. >> >> Just out of curiosity how many of those messages are being passed >> to SEC? > > I actually have a complex logging infrastructure, I have the syslog > messages being sent to different farms of machines for archiving, > database > inserts (ad-hoc querying), and alerting. > > on the SEC alerting system I have rsyslog do filtering on > programname and > then send different programnames to different instances of SEC (so > that > each instance of SEC only sees fairly relavent logs) > > I haven't checked recently to see what the total log volume going to > SEC > is (or what the peak CPU utilization of SEC is). actually, looking > at it > I do have one instance that is seeing every syslog message, but that > instance only has two rules in it. that instance is using about the > same > CPU as rsyslog is, the next busiest instance is useing about 1/5 of > that > > the peak messages/sec tests mentioned above were to receive and > write to > disk. I expect that writing to a pipe would be the same, doing more > complex things would probably slow it down. > > David Lang > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart > your > developing skills, take BlackBerry mobile applications to market and > stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Simple-evcorr-users mailing list > Simple-evcorr-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
