On Fri, 3 May 2013, John Zhang wrote:
Hi everyone,I am researching the big data security log management, such as Kibana + ElasticSearch + Logstash for my security log management, I need event correlation on this platform, i know SEC( http://simple-evcorr.sourceforge.net/) can do event correlation. Do you have any idea of SEC on such big data security log platform? Any experience, any reference? Any comment, advise will be highly appreciated!
my presentation on logging from LISA is at https://www.usenix.org/conference/lisa12/building-100k-logsec-logging-infrastructure
I used rsyslog as my transport, but if logstash can handle your load, there's nothing wrong with using it.
What sort of data volume are you expecting?SEC works wonderfully well for this sort of thing, but the one issue that you will run into is the fact that SEC is single-threaded, so it's limited to how much data a single process can handle (how much depends on the number and complexity of your ruleset).
The work-around for this is to split your logs into seperate feeds and have a seperate copy of SEC processing each feed. If you can split the feeds so that your correlation rules don't need to be across feeds, it's really easy to do this. Also note that you don't need to run all the feeds on one system.
If you do need to correlate across feeds, take a look at this paper from a few years ago http://static.usenix.org/events/lisa10/tech/full_papers/Krizak.pdf they used syslog-ng and their own analysis engine, but the approach of having small instances that look at a particular type of log and then when one instance finds something, instead of alerting directly, it generates a new log message and there is a instance of SEC that only watches these 'internal' messages and that instance generates all the alerts, and deals with correlation rules that need to take into account the different feeds.
David Lang
------------------------------------------------------------------------------ Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
------------------------------------------------------------------------------ Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
