Ah, ok. thanks for clarifying!
______________________________________________________________
Clayton Dukes
______________________________________________________________
On Mon, Feb 11, 2013 at 6:25 PM, Busko, Steve <sbu...@cogentco.com> wrote:
> Clayton,****
>
> ** **
>
> We did not need RabbitMQ for helping with the quantity of inserts. I only
> provided the number show how much we were pushing through it in case anyone
> wanted a reference. We opted to add MQ for the ability to control and
> monitor the data via its queues instead of having other parts of the system
> deal with it. ****
>
> ** **
>
> One case we enabled an ACK on the output queue (from SEC to DB) so that we
> don’t lose events should the script handling the DB insertion fail. MQ
> would queue up the non-ACK’d entries (which can also be easily measured)
> until the issue resolved. It’s use all depends on your needs, we just found
> RabbitMQ filled our needs more than the raw pipe.****
>
> ** **
>
> **- **Steve****
>
> ** **
>
> *From:* Clayton Dukes [mailto:cdu...@gmail.com]
> *Sent:* Monday, February 11, 2013 4:05 PM
> *To:* Busko, Steve
> *Cc:* Risto Vaarandi; Simple-evcorr-users
> *Subject:* Re: [Simple-evcorr-users] extending output types (RabbitMQ)****
>
> ** **
>
> Hi Steve,****
>
> I'm a little confused. Why would you need RabbitMQ to insert only 40
> events/sec?****
>
> My syslog tool (LogZilla) does 15k events/sec into MySQL without using
> Rabbit.****
>
> We are working now (using RabbitMQ) on getting that number to around 120k
> eps.****
>
> ** **
>
> P.S.****
>
> I would love it if SEC could process at that rate :-)****
>
>
> ****
>
> ______________________________________________________________
>
> Clayton Dukes
> ______________________________________________________________****
>
> ** **
>
> On Mon, Feb 11, 2013 at 3:12 PM, Busko, Steve <sbu...@cogentco.com> wrote:
> ****
>
> Hello Ristro,
>
> While I don't have specific implementation feedback on this topic, I did
> want to throw in that, like Gary, we are also using RabbitMQ and would like
> to see sec integrate in some way possibly affecting this thread in part.
>
> Initially we used pipes, but found them not robust enough. So we tried
> RabbitMQ and have been much happier with the flexibility and control. We
> have been running it for around 6 months in a development environment
> taking in around low 40s msgs/sec from a single syslog source and outputs
> slightly less than half that into a single DB. In our case we didn't do
> anything fancy to integrate, just inserted the MQ perl code/functionality
> directly into sec and created an additional "queue" command to use vs. a
> write.
>
> So we would like to support both the use of RabbitMQ and the consideration
> of integrating it (or as a plug-in) with sec.
>
> Regards,
> Steve Busko | NOC Tools Manager
>
> > -----Original Message-----
> > From: Risto Vaarandi [mailto:risto.vaara...@seb.ee]
> > Sent: Wednesday, February 06, 2013 12:38 PM
> > To: Simple-evcorr-users
> > Subject: [Simple-evcorr-users] extending output types
> >
> > hi all,
> >
> > as you know, currently there are number of outputs which are not
> > directly supported by sec. For example, with 'pipe' action one can feed
> > only one event to an external program (with most programs exiting after
> > the 'pipe' action closes the pipe). Also, the 'write' action opens a
> > given file before each access and closes it afterwards. While this makes
> > output file rotation worry-free, it is not very efficient. Finally, sec
> > does not directly support various sockets as output (for example, it is
> > not possible to connect directly to a syslog server on another host and
> > issue syslog events without any intermediaries).
> >
> > Therefore, would there be any interest to an output action which would
> > leave the output open/running/connected, so that the next access of the
> > same output would not have to do reopen/restart/connect again? This
> > action would have to support a number of output types, e.g.,
> >
> > write2 udp myserver:514 This is my event
> > write2 tcp myserver:514 Another event
> > write2 file /var/log/myevents Third event
> > write2 program /bin/logger -p user.notice Fourth event
> >
> > It is probably easy to see that with sockets and files it is not that
> > hard to distinguish them from each other, since the filename or server
> > name plus port number can serve as unique identifiers. With command
> > lines, however, it is somewhat more difficult, since an extra space or
> > different ordering of options makes two command lines different, even if
> > their effect is exactly the same. Also, since sec uses positional
> > parameters, there is an issue with parsing, although this could be
> > solved with the use of (). In addition, one workaround for both issues
> > would be storing command line in an action list variable.
> > The action outlines above would also need an output rotation mechanism,
> > but this could be implemented with adding this to SIGUSR2 handler
> > (closing all currently open outputs).
> >
> > Would the action above be of interest, and are there any other ideas for
> > handling these situations?
> >
> > kind regards,
> > risto
> >
> >
> ------------------------------------------------------------------------------
> > Free Next-Gen Firewall Hardware Offer
> > Buy your Sophos next-gen firewall before the end March 2013
> > and get the hardware for free! Learn more.
> > http://p.sf.net/sfu/sophos-d2d-feb
> > _______________________________________________
> > Simple-evcorr-users mailing list
> > Simple-evcorr-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users****
>
> ** **
>
------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
