Aye! Agn, bad assumption on my part. I assumed syslog-ng would queue while the receiving end was offline. Whatever msgs didn't fit into the queue would drop onto the floor.
Time for me to reread the docs. No Cisco in the pipeline, and the last msg storm was brought about by an errant ldap configuration. Tnx --joe Sent from Earth via a portable device. On Oct 24, 2009, at 0:23, Clayton Dukes <cdu...@gmail.com> wrote: > The syslog instance sending the messages won't requeue them (unless > you've done something special). Since syslog uses UDP by default, > there's no way for the sender to know that the messages ever reached > their destination. > If you want to throttle, the only real way I see to do that is to > either use disk-based buffering or implement rate limiting (assuming > you're using cisco devices). > If you just wanted to drop floods, I think SEC can do that by > detecting subsequent messages from the same host within the same > timeframe and dump the ones that match (not positive though), but then > you still have the problem of possibly missing something important > like a machine screaming for attention :-) > > > > > On Fri, Oct 23, 2009 at 10:43 PM, J Carvalho <dio...@rmws.net> wrote: >> Your time and efforts are much appreciated. >> Different tack. >> >> Instead of condensing the msgs, I'm thinking of initiating a snort- >> style >> iptable/ipfilter BLOCK against the machine flooding my collector. >> The syslog instance on the client machine will queue the msgs until >> the >> server is once again willing to listen. >> I was tempted to do this using snort and snortsam, but I'm thinking >> SEC and >> syslog(ng) may be a bit easier. >> >> Thoughts or suggestions? >> --joe >> >> >> On Oct 23, 2009, at 09:38:30:EDT, Clayton Dukes wrote: >> >>> You can "compress" the messages using the deduplication >>> functionality of >>> php-syslog-ng so that only a single row is stored in the db with a >>> count of >>> 10,000 vs 10,000 separate rows with the same message. >>> >>> The problem u face with the flood of messages is still going to >>> give u >>> trouble though unless u run the pro version of syslog-ng which does >>> disk-based buffering. Another bottleneck is using pipes which are >>> somewhat >>> limited in the amount of incoming messages u can process. I've >>> worked very >>> hard on trying to get php-syslog-ng to scale beyond 1k mps but >>> have been >>> unsuccessful thus far. To that end, we're doing a complete rewrite >>> of the >>> code (now in alpha) which can process roughly 50k mps and can >>> query 300 >>> million rows in just under 4 seconds. Unfortunately this doesn't >>> help u >>> right now though :-) >>> >>> Sent from my iPhone >>> >>> On Oct 22, 2009, at 10:04 PM, J Carvalho <dio...@rmws.net> wrote: >>> >>>> I'm in the process of protecting my loghosts running syslog-ng. >>>> >>>> syslog-ng injects the msgs into mysql. The msgs are parsed into >>>> their >>>> appropriate fields prior to db injection. >>>> I can maintain a steady throughput of 3k msgs/sec. >>>> >>>> I ran into a problem a fews week back. A syslogging host went off >>>> the >>>> rails and pushed more msgs than I could process. >>>> Msgs spewed in, I couldn't drain the queue fast enough, mysql >>>> consumed >>>> more memory than we should've allotted, fire fell from the sky. >>>> >>>> I've got snort/snortsam in place as part of a brute force syslog >>>> DOS >>>> protection mechanism. If msgs come to the door, faster than can be >>>> handled, a BLOCK is put into place. >>>> >>>> I wanted to compress the msgs in order to eliminate writing 10k >>>> duplicate msgs...or the useless "last message repeated n times", in >>>> the hope of writing a single message with start/stop times and # of >>>> msgs seen in that timeframe. >>>> >>>> I believe I have sufficient cpu and memory(X6250) with a 10Gb feed, >>>> unfortunately the long pole are SATA 500GB 8M cache drives. >>>> >>>> The iptable BLOCK is a last resort to protect the collector from >>>> the >>>> forces of evil. >>>> tnx >>>> --joe >>>> >>>> >>>> On Oct 22, 2009, at 16:15:07:EDT, da...@lang.hm wrote: >>>> >>>>> On Thu, 22 Oct 2009, John P. Rouillard wrote: >>>>> >>>>>> In message <alpine.deb.2.00.0910221240160.11...@asgard.lang.hm>, >>>>>> da...@lang.hm writes: >>>>>>> >>>>>>> On Thu, 22 Oct 2009, Clayton Dukes wrote: >>>>>>>> >>>>>>>> If you're using the latest version of php-syslog-ng it has a >>>>>>>> built in >>>>>>>> deduplication function. >>>>>>>> http://code.google.com/p/php-syslog-ng >>>>>>> >>>>>>> most syslog daemons have the ability to say 'last message >>>>>>> repeated X >>>>>>> times' >>>>>>> >>>>>>> rsyslog has an option to include the message that was repeated >>>>>>> at >>>>>>> the end >>>>>>> of the 'last message repeated' line >>>>>> >>>>>> Yeah I usually disable the "last message repeated X times" first >>>>>> thing >>>>>> 8-). However if you are getting multiple events/second then it's >>>>>> usually easy enough to work around. >>>>>> >>>>>>> I have rsyslog handling tens of thousands of messages/sec >>>>>>> currently with >>>>>>> fairly low cpu utilization (<20% of one core) and tested it >>>>>>> about 6 >>>>>>> months ago up to ~80K messages/sec sustained, >300K messages/sec >>>>>>> peak >>>>>>> (essentially wire speed of Gig-E). the current dev version has >>>>>>> significant improvements that should allow this to go >>>>>>> significantly >>>>>>> higher, but the kinks are still being worked out of it. >>>>>> >>>>>> Just out of curiosity how many of those messages are being passed >>>>>> to SEC? >>>>> >>>>> I actually have a complex logging infrastructure, I have the >>>>> syslog >>>>> messages being sent to different farms of machines for archiving, >>>>> database >>>>> inserts (ad-hoc querying), and alerting. >>>>> >>>>> on the SEC alerting system I have rsyslog do filtering on >>>>> programname and >>>>> then send different programnames to different instances of SEC (so >>>>> that >>>>> each instance of SEC only sees fairly relavent logs) >>>>> >>>>> I haven't checked recently to see what the total log volume >>>>> going to >>>>> SEC >>>>> is (or what the peak CPU utilization of SEC is). actually, looking >>>>> at it >>>>> I do have one instance that is seeing every syslog message, but >>>>> that >>>>> instance only has two rules in it. that instance is using about >>>>> the >>>>> same >>>>> CPU as rsyslog is, the next busiest instance is useing about 1/5 >>>>> of >>>>> that >>>>> >>>>> the peak messages/sec tests mentioned above were to receive and >>>>> write to >>>>> disk. I expect that writing to a pipe would be the same, doing >>>>> more >>>>> complex things would probably slow it down. >>>>> >>>>> David Lang >>>>> >>>>> >>>>> --- >>>>> --- >>>>> --- >>>>> --- >>>>> ------------------------------------------------------------------ >>>>> Come build with us! The BlackBerry(R) Developer Conference in >>>>> SF, CA >>>>> is the only developer event you need to attend this year. >>>>> Jumpstart >>>>> your >>>>> developing skills, take BlackBerry mobile applications to market >>>>> and >>>>> stay >>>>> ahead of the curve. Join us from November 9 - 12, 2009. Register >>>>> now! >>>>> http://p.sf.net/sfu/devconference >>>>> _______________________________________________ >>>>> Simple-evcorr-users mailing list >>>>> Simple-evcorr-users@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >>>> >>>> >>>> >>>> --- >>>> --- >>>> --- >>>> --- >>>> ------------------------------------------------------------------ >>>> Come build with us! The BlackBerry(R) Developer Conference in SF, >>>> CA >>>> is the only developer event you need to attend this year. >>>> Jumpstart your >>>> developing skills, take BlackBerry mobile applications to market >>>> and stay >>>> ahead of the curve. Join us from November 9 - 12, 2009. Register >>>> now! >>>> http://p.sf.net/sfu/devconference >>>> _______________________________________________ >>>> Simple-evcorr-users mailing list >>>> Simple-evcorr-users@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >> >> > > > > -- > ______________________________________________________________ > > Clayton Dukes > ______________________________________________________________ > > --- > --- > --- > --------------------------------------------------------------------- > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart > your > developing skills, take BlackBerry mobile applications to market and > stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Simple-evcorr-users mailing list > Simple-evcorr-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
