In message <CALD8BLSZxK-JsM5Gkg56GsbZ-URx3EEuw2beqYJmP=y4bpm...@mail.gmail.com>, John Zhang writes:
>My requirements are also collecting logs from network, server and >applications, their volumes are about 50k log/s; except searching, also >need event correlation, alerting. > >I am not sure if SEC can handle the above log volume, and how is >architecture can handle such load? You would want to split your feed into multiple parts/streams and have separate SEC processes monitor each stream. http://www.cs.umb.edu/~rouilj/sec/sec_paper_full.pdf section 4 still has good advice on making SEC perform well even 9 years later. Section 4.2 desribes how to run multiple sets of rules in parallel by having a master SEC process start child SEC processes. Section 4.3 sketches how to coordinate multiple SEC processes on different hosts. I had 5 SEC running across three nodes all working together. Whether this is worth the extra complexity in todays multi-core environment is an open question. The child SEC processes handle filtering/correlation for the events they can see and pass events that need to be correlated with events from other event streams to the parent SEC process. I had 4 or 5 child sec processes handling approx. 10k events per second in my lab in early 2004 or so. Given the advancements in speed of CPU's and the ability to spawn more processes on multi-core systems I see no reason 50K events/sec couldn't be achieved for a suitable segmentation of your event stream. One thing SEC is weak on is performance metrics and figuring out how to tune a ruleset for best performance. This is still a black art. Given the number of events/sec you need to process you may need to do a good amount of performance tuning. -- -- rouilj John Rouillard =========================================================================== My employers don't acknowledge my existence much less my opinions. ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. This 200-page book is written by three acclaimed leaders in the field. The early access version is available now. Download your free book today! http://p.sf.net/sfu/neotech_d2d_may _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
