Your time and efforts are much appreciated. Different tack. Instead of condensing the msgs, I'm thinking of initiating a snort- style iptable/ipfilter BLOCK against the machine flooding my collector. The syslog instance on the client machine will queue the msgs until the server is once again willing to listen. I was tempted to do this using snort and snortsam, but I'm thinking SEC and syslog(ng) may be a bit easier.
Thoughts or suggestions? --joe On Oct 23, 2009, at 09:38:30:EDT, Clayton Dukes wrote: > You can "compress" the messages using the deduplication > functionality of php-syslog-ng so that only a single row is stored > in the db with a count of 10,000 vs 10,000 separate rows with the > same message. > > The problem u face with the flood of messages is still going to give > u trouble though unless u run the pro version of syslog-ng which > does disk-based buffering. Another bottleneck is using pipes which > are somewhat limited in the amount of incoming messages u can > process. I've worked very hard on trying to get php-syslog-ng to > scale beyond 1k mps but have been unsuccessful thus far. To that > end, we're doing a complete rewrite of the code (now in alpha) which > can process roughly 50k mps and can query 300 million rows in just > under 4 seconds. Unfortunately this doesn't help u right now > though :-) > > Sent from my iPhone > > On Oct 22, 2009, at 10:04 PM, J Carvalho <dio...@rmws.net> wrote: > >> I'm in the process of protecting my loghosts running syslog-ng. >> >> syslog-ng injects the msgs into mysql. The msgs are parsed into their >> appropriate fields prior to db injection. >> I can maintain a steady throughput of 3k msgs/sec. >> >> I ran into a problem a fews week back. A syslogging host went off the >> rails and pushed more msgs than I could process. >> Msgs spewed in, I couldn't drain the queue fast enough, mysql >> consumed >> more memory than we should've allotted, fire fell from the sky. >> >> I've got snort/snortsam in place as part of a brute force syslog DOS >> protection mechanism. If msgs come to the door, faster than can be >> handled, a BLOCK is put into place. >> >> I wanted to compress the msgs in order to eliminate writing 10k >> duplicate msgs...or the useless "last message repeated n times", in >> the hope of writing a single message with start/stop times and # of >> msgs seen in that timeframe. >> >> I believe I have sufficient cpu and memory(X6250) with a 10Gb feed, >> unfortunately the long pole are SATA 500GB 8M cache drives. >> >> The iptable BLOCK is a last resort to protect the collector from the >> forces of evil. >> tnx >> --joe >> >> >> On Oct 22, 2009, at 16:15:07:EDT, da...@lang.hm wrote: >> >>> On Thu, 22 Oct 2009, John P. Rouillard wrote: >>> >>>> In message <alpine.deb.2.00.0910221240160.11...@asgard.lang.hm>, >>>> da...@lang.hm writes: >>>>> On Thu, 22 Oct 2009, Clayton Dukes wrote: >>>>>> If you're using the latest version of php-syslog-ng it has a >>>>>> built in >>>>>> deduplication function. >>>>>> http://code.google.com/p/php-syslog-ng >>>>> >>>>> most syslog daemons have the ability to say 'last message >>>>> repeated X >>>>> times' >>>>> >>>>> rsyslog has an option to include the message that was repeated at >>>>> the end >>>>> of the 'last message repeated' line >>>> >>>> Yeah I usually disable the "last message repeated X times" first >>>> thing >>>> 8-). However if you are getting multiple events/second then it's >>>> usually easy enough to work around. >>>> >>>>> I have rsyslog handling tens of thousands of messages/sec >>>>> currently with >>>>> fairly low cpu utilization (<20% of one core) and tested it >>>>> about 6 >>>>> months ago up to ~80K messages/sec sustained, >300K messages/sec >>>>> peak >>>>> (essentially wire speed of Gig-E). the current dev version has >>>>> significant improvements that should allow this to go >>>>> significantly >>>>> higher, but the kinks are still being worked out of it. >>>> >>>> Just out of curiosity how many of those messages are being passed >>>> to SEC? >>> >>> I actually have a complex logging infrastructure, I have the syslog >>> messages being sent to different farms of machines for archiving, >>> database >>> inserts (ad-hoc querying), and alerting. >>> >>> on the SEC alerting system I have rsyslog do filtering on >>> programname and >>> then send different programnames to different instances of SEC (so >>> that >>> each instance of SEC only sees fairly relavent logs) >>> >>> I haven't checked recently to see what the total log volume going to >>> SEC >>> is (or what the peak CPU utilization of SEC is). actually, looking >>> at it >>> I do have one instance that is seeing every syslog message, but that >>> instance only has two rules in it. that instance is using about the >>> same >>> CPU as rsyslog is, the next busiest instance is useing about 1/5 of >>> that >>> >>> the peak messages/sec tests mentioned above were to receive and >>> write to >>> disk. I expect that writing to a pipe would be the same, doing more >>> complex things would probably slow it down. >>> >>> David Lang >>> >>> ------------------------------------------------------------------------------ >>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA >>> is the only developer event you need to attend this year. Jumpstart >>> your >>> developing skills, take BlackBerry mobile applications to market and >>> stay >>> ahead of the curve. Join us from November 9 - 12, 2009. Register >>> now! >>> http://p.sf.net/sfu/devconference >>> _______________________________________________ >>> Simple-evcorr-users mailing list >>> Simple-evcorr-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >> >> >> ------------------------------------------------------------------------------ >> Come build with us! The BlackBerry(R) Developer Conference in SF, CA >> is the only developer event you need to attend this year. Jumpstart >> your >> developing skills, take BlackBerry mobile applications to market >> and stay >> ahead of the curve. Join us from November 9 - 12, 2009. Register now! >> http://p.sf.net/sfu/devconference >> _______________________________________________ >> Simple-evcorr-users mailing list >> Simple-evcorr-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
