Hi David,
Thanks!
I am very interested in your solution, could you provide more details?
My requirements are also collecting logs from network, server and
applications, their volumes are about 50k log/s; except searching, also
need event correlation, alerting.
I am not sure if SEC can handle the above log volume, and how is
architecture can handle such load?
Thanks!
Best regards,
John
在 2013年5月5日星期日，David Lang 写道：
> On Fri, 3 May 2013, John Zhang wrote:
>
> Hi everyone,
>>
>> I am researching the big data security log management, such as Kibana +
>> ElasticSearch + Logstash for my security log management, I need event
>> correlation on this platform, i know SEC(
>> http://simple-evcorr.**sourceforge.net/<http://simple-evcorr.sourceforge.net/>)
>> can do event correlation.
>>
>> Do you have any idea of SEC on such big data security log platform? Any
>> experience, any reference?
>>
>> Any comment, advise will be highly appreciated!
>>
>
> my presentation on logging from LISA is at https://www.usenix.org/**
> conference/lisa12/building-**100k-logsec-logging-**infrastructure<https://www.usenix.org/conference/lisa12/building-100k-logsec-logging-infrastructure>
>
> I used rsyslog as my transport, but if logstash can handle your load,
> there's nothing wrong with using it.
>
> What sort of data volume are you expecting?
>
> SEC works wonderfully well for this sort of thing, but the one issue that
> you will run into is the fact that SEC is single-threaded, so it's limited
> to how much data a single process can handle (how much depends on the
> number and complexity of your ruleset).
>
> The work-around for this is to split your logs into seperate feeds and
> have a seperate copy of SEC processing each feed. If you can split the
> feeds so that your correlation rules don't need to be across feeds, it's
> really easy to do this. Also note that you don't need to run all the feeds
> on one system.
>
> If you do need to correlate across feeds, take a look at this paper from a
> few years ago http://static.usenix.org/**events/lisa10/tech/full_**
> papers/Krizak.pdf<http://static.usenix.org/events/lisa10/tech/full_papers/Krizak.pdf>they
> used syslog-ng and their own analysis engine, but the approach of
> having small instances that look at a particular type of log and then when
> one instance finds something, instead of alerting directly, it generates a
> new log message and there is a instance of SEC that only watches these
> 'internal' messages and that instance generates all the alerts, and deals
> with correlation rules that need to take into account the different feeds.
>
> David Lang
>
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and
their applications. This 200-page book is written by three acclaimed
leaders in the field. The early access version is available now.
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
