On Tue, 7 May 2013, John Zhang wrote: > Hi David, > > Thanks! > > I am very interested in your solution, could you provide more details? > > My requirements are also collecting logs from network, server and > applications, their volumes are about 50k log/s; except searching, also > need event correlation, alerting. > > I am not sure if SEC can handle the above log volume, and how is > architecture can handle such load?
>> my presentation on logging from LISA is at >> https://www.usenix.org/conference/lisa12/building-**100k-logsec-logging-**infrastructure<https://www.usenix.org/conference/lisa12/building-100k-logsec-logging-infrastructure> A single instance of SEC is probably not going to be able to handle that volume of logs. However, if you think about the correlations that you are doing, very few of them involve different types of logs. In my rsyslog.conf, I have a series of rules that look like: :fromhost-ip, isequal, "127.0.0.1" @192.168.210.218;RSYSLOG_TraditionalForwardFormat & /var/log/messages;RSYSLOG_TraditionalFileFormat & ~ :programname, isequal, "vmstat" |/var/fifo/vmstat;RSYSLOG_TraditionalFileFormat & ~ :programname, isequal, "iaalog" |/var/fifo/iaalog;RSYSLOG_TraditionalFileFormat & ~ :programname, isequal, "snort" /var/log/snort;RSYSLOG_TraditionalFileFormat & ~ :programname, isequal, "nsmnow" |/var/fifo/nsmnow;RSYSLOG_TraditionalFileFormat & ~ #:programname, isequal, "ossec" /var/log/ossec;RSYSLOG_TraditionalFileFormat & ~ :rawmsg, contains, " netips-" |/var/fifo/tippingpoint;RSYSLOG_TraditionalFileFormat & ~ # filter out some high volume log entries that we don't have any rules for # added 6-12-12 by David Lang due to the fact that the syslog instance of sec was not keeping up with the log volume # :programname, isequal, "getprocs" ~ :programname, isequal, "iostat" ~ *.* |/var/fifo/syslog;RSYSLOG_TraditionalFileFormat This causes rsyslog to split the logs into several, much smaller feeds, with each feed being sent a different named pipe (created with mkfifo) I then have several instances of SEC running, each one reading a particular named pipe. Since each instance of SEC is only looking at a subset of the logs, it only needs the rules related to that subset. This improved performance in two ways. 1. that instance of SEC is processing fewer lines of logs 2. that instance of SEC is going through fewer rules for each log line that it processes. In my setup, I can get away with just splitting the logs based on the programname, but if you find that the result is still too large, you can create a more complex filter and further subdivide the log feeds. The complication with this approach is when you want to setup a situation where you set a context based on one type of log message and then want to refer to that context in rules related to a different type of log message. The way you deal with this is to have the instance of SEC that finds something that you want other instances to care about log a message that some other instance will see. On a single machine, you can follow the approach that Risto describes where you have one master instance of SEC that doesn't actually look at logfiles, it just spawns multiple other instances of SEC, each configured to look at a subset of logfiles, then these instances can write output which is read by the master instance (which can then combine the contexts to do alerting) If you end up needing to split your logs so much that the resulting instances of SEC cannot fit on a single machine, or if you don't want to have one instance of SEC spawn the others, or if you have more cross-communication than this approach provides, you can go the next step to the general approach described in this paper ( http://static.usenix.org/events/lisa10/tech/full_papers/Krizak.pdf ) where the communication is achieved by having one instance of the log analysis tool create a new log message that gets read by other instances. In this case, they also had one master instance that did all the alerting, but the same approach could be used for one instance to generate a log that another instance reads and uses to set a context. David Lang ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. This 200-page book is written by three acclaimed leaders in the field. The early access version is available now. Download your free book today! http://p.sf.net/sfu/neotech_d2d_may _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
