On 02/11/2013 11:04 PM, Clayton Dukes wrote: > Hi Steve, > I'm a little confused. Why would you need RabbitMQ to insert only 40 > events/sec? > My syslog tool (LogZilla) does 15k events/sec into MySQL without using > Rabbit. > We are working now (using RabbitMQ) on getting that number to around > 120k eps. > > P.S. > I would love it if SEC could process at that rate :-)
If you would like to go up to event rates like 100-200k and do simple processing for events (write them immediately into some storage in large batches, without doing any time based analysis), I would say you would need a different tool for this. If I would have to write something like this myself, I'd definitely do it in C with a few simpler commandline/configfile options. For example, for my own database logging, I am currently using rsyslog which is extremely fast for handling heavy message loads. Since SEC is a perl based tool, there is no chance for it to compete with C-based binaries when it comes to fast "accept->filter->store" schemes. Nevertheless, its main focus is elsewhere -- its for keeping a lot of state information and custom data structures in memory, and doing a lot of time based analysis where extended functionality of perl is very handy. If you don't need such functionality and just want to write syslog-events into databases in large batches, rsyslog or syslog-ng are probably much better options. kind regards, risto > > ______________________________________________________________ > > Clayton Dukes > ______________________________________________________________ > > > On Mon, Feb 11, 2013 at 3:12 PM, Busko, Steve <sbu...@cogentco.com > <mailto:sbu...@cogentco.com>> wrote: > > Hello Ristro, > > While I don't have specific implementation feedback on this topic, I > did want to throw in that, like Gary, we are also using RabbitMQ and > would like to see sec integrate in some way possibly affecting this > thread in part. > > Initially we used pipes, but found them not robust enough. So we > tried RabbitMQ and have been much happier with the flexibility and > control. We have been running it for around 6 months in a > development environment taking in around low 40s msgs/sec from a > single syslog source and outputs slightly less than half that into a > single DB. In our case we didn't do anything fancy to integrate, > just inserted the MQ perl code/functionality directly into sec and > created an additional "queue" command to use vs. a write. > > So we would like to support both the use of RabbitMQ and the > consideration of integrating it (or as a plug-in) with sec. > > Regards, > Steve Busko | NOC Tools Manager > ------------------------------------------------------------------------------ Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
