On Thu, 22 Oct 2009, Clayton Dukes wrote:
John,
If you're using the latest version of php-syslog-ng it has a built in
deduplication function.
http://code.google.com/p/php-syslog-ng
most syslog daemons have the ability to say 'last message repeated X
times'
rsyslog has an option to include the message that was repeated at the end
of the 'last message repeated' line
I have rsyslog handling tens of thousands of messages/sec currently with
fairly low cpu utilization (<20% of one core) and tested it about 6
months ago up to ~80K messages/sec sustained, >300K messages/sec peak
(essentially wire speed of Gig-E). the current dev version has
significant improvements that should allow this to go significantly
higher, but the kinks are still being worked out of it.
how big is this flood you are seeing?
David Lang
Integrating php-syslog-ng with SEC is quite simple, but let me know if
you need help.
On Thu, Oct 22, 2009 at 1:55 PM, John P. Rouillard <rou...@cs.umb.edu> wrote:
In message <76a8b853-9c9d-47bc-b610-aa6aad0f2...@rmws.net>,
J Carvalho writes:
I'd like to use sec to condense incoming syslog events. I had a system
generate thousands of msgs per second and the result was a swamped
syslog collector.
Would it be possible to use SEC to:
1. read the input stream
2. pass msgs to the syslog file until it sees a msg storm
3. condense the msgs based on time or msg count to prevent swamping
syslog
4. write a msg to the syslog file with a count of msgs processed
during the 'storm'.
5. continue processing input stream.
Maybe I am missing something here, but where is the input stream in 1
comming from? It sounds like some application is using the syslog api
to send data to syslog directly and flooding it. There is no easy way
to get SEC in between the application and syslog.
Also while SEC can be convinced to handle thousands of messages/sec
the cpu usage would almost certainly be more than what syslog uses to
dump the messages to disk.
--
-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
