The syslog instance sending the messages won't requeue them (unless you've done something special). Since syslog uses UDP by default, there's no way for the sender to know that the messages ever reached their destination. If you want to throttle, the only real way I see to do that is to either use disk-based buffering or implement rate limiting (assuming you're using cisco devices). If you just wanted to drop floods, I think SEC can do that by detecting subsequent messages from the same host within the same timeframe and dump the ones that match (not positive though), but then you still have the problem of possibly missing something important like a machine screaming for attention :-)
On Fri, Oct 23, 2009 at 10:43 PM, J Carvalho <dio...@rmws.net> wrote: > Your time and efforts are much appreciated. > Different tack. > > Instead of condensing the msgs, I'm thinking of initiating a snort-style > iptable/ipfilter BLOCK against the machine flooding my collector. > The syslog instance on the client machine will queue the msgs until the > server is once again willing to listen. > I was tempted to do this using snort and snortsam, but I'm thinking SEC and > syslog(ng) may be a bit easier. > > Thoughts or suggestions? > --joe > > > On Oct 23, 2009, at 09:38:30:EDT, Clayton Dukes wrote: > >> You can "compress" the messages using the deduplication functionality of >> php-syslog-ng so that only a single row is stored in the db with a count of >> 10,000 vs 10,000 separate rows with the same message. >> >> The problem u face with the flood of messages is still going to give u >> trouble though unless u run the pro version of syslog-ng which does >> disk-based buffering. Another bottleneck is using pipes which are somewhat >> limited in the amount of incoming messages u can process. I've worked very >> hard on trying to get php-syslog-ng to scale beyond 1k mps but have been >> unsuccessful thus far. To that end, we're doing a complete rewrite of the >> code (now in alpha) which can process roughly 50k mps and can query 300 >> million rows in just under 4 seconds. Unfortunately this doesn't help u >> right now though :-) >> >> Sent from my iPhone >> >> On Oct 22, 2009, at 10:04 PM, J Carvalho <dio...@rmws.net> wrote: >> >>> I'm in the process of protecting my loghosts running syslog-ng. >>> >>> syslog-ng injects the msgs into mysql. The msgs are parsed into their >>> appropriate fields prior to db injection. >>> I can maintain a steady throughput of 3k msgs/sec. >>> >>> I ran into a problem a fews week back. A syslogging host went off the >>> rails and pushed more msgs than I could process. >>> Msgs spewed in, I couldn't drain the queue fast enough, mysql consumed >>> more memory than we should've allotted, fire fell from the sky. >>> >>> I've got snort/snortsam in place as part of a brute force syslog DOS >>> protection mechanism. If msgs come to the door, faster than can be >>> handled, a BLOCK is put into place. >>> >>> I wanted to compress the msgs in order to eliminate writing 10k >>> duplicate msgs...or the useless "last message repeated n times", in >>> the hope of writing a single message with start/stop times and # of >>> msgs seen in that timeframe. >>> >>> I believe I have sufficient cpu and memory(X6250) with a 10Gb feed, >>> unfortunately the long pole are SATA 500GB 8M cache drives. >>> >>> The iptable BLOCK is a last resort to protect the collector from the >>> forces of evil. >>> tnx >>> --joe >>> >>> >>> On Oct 22, 2009, at 16:15:07:EDT, da...@lang.hm wrote: >>> >>>> On Thu, 22 Oct 2009, John P. Rouillard wrote: >>>> >>>>> In message <alpine.deb.2.00.0910221240160.11...@asgard.lang.hm>, >>>>> da...@lang.hm writes: >>>>>> >>>>>> On Thu, 22 Oct 2009, Clayton Dukes wrote: >>>>>>> >>>>>>> If you're using the latest version of php-syslog-ng it has a >>>>>>> built in >>>>>>> deduplication function. >>>>>>> http://code.google.com/p/php-syslog-ng >>>>>> >>>>>> most syslog daemons have the ability to say 'last message repeated X >>>>>> times' >>>>>> >>>>>> rsyslog has an option to include the message that was repeated at >>>>>> the end >>>>>> of the 'last message repeated' line >>>>> >>>>> Yeah I usually disable the "last message repeated X times" first >>>>> thing >>>>> 8-). However if you are getting multiple events/second then it's >>>>> usually easy enough to work around. >>>>> >>>>>> I have rsyslog handling tens of thousands of messages/sec >>>>>> currently with >>>>>> fairly low cpu utilization (<20% of one core) and tested it about 6 >>>>>> months ago up to ~80K messages/sec sustained, >300K messages/sec >>>>>> peak >>>>>> (essentially wire speed of Gig-E). the current dev version has >>>>>> significant improvements that should allow this to go significantly >>>>>> higher, but the kinks are still being worked out of it. >>>>> >>>>> Just out of curiosity how many of those messages are being passed >>>>> to SEC? >>>> >>>> I actually have a complex logging infrastructure, I have the syslog >>>> messages being sent to different farms of machines for archiving, >>>> database >>>> inserts (ad-hoc querying), and alerting. >>>> >>>> on the SEC alerting system I have rsyslog do filtering on >>>> programname and >>>> then send different programnames to different instances of SEC (so >>>> that >>>> each instance of SEC only sees fairly relavent logs) >>>> >>>> I haven't checked recently to see what the total log volume going to >>>> SEC >>>> is (or what the peak CPU utilization of SEC is). actually, looking >>>> at it >>>> I do have one instance that is seeing every syslog message, but that >>>> instance only has two rules in it. that instance is using about the >>>> same >>>> CPU as rsyslog is, the next busiest instance is useing about 1/5 of >>>> that >>>> >>>> the peak messages/sec tests mentioned above were to receive and >>>> write to >>>> disk. I expect that writing to a pipe would be the same, doing more >>>> complex things would probably slow it down. >>>> >>>> David Lang >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA >>>> is the only developer event you need to attend this year. Jumpstart >>>> your >>>> developing skills, take BlackBerry mobile applications to market and >>>> stay >>>> ahead of the curve. Join us from November 9 - 12, 2009. Register now! >>>> http://p.sf.net/sfu/devconference >>>> _______________________________________________ >>>> Simple-evcorr-users mailing list >>>> Simple-evcorr-users@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA >>> is the only developer event you need to attend this year. Jumpstart your >>> developing skills, take BlackBerry mobile applications to market and stay >>> ahead of the curve. Join us from November 9 - 12, 2009. Register now! >>> http://p.sf.net/sfu/devconference >>> _______________________________________________ >>> Simple-evcorr-users mailing list >>> Simple-evcorr-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users > > -- ______________________________________________________________ Clayton Dukes ______________________________________________________________ ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
