hi Eric, these features are implemented in a better way in the most recent beta version which I have currently under testing. Previously, connecting an external program persistently was only possible with first starting the program with the 'spawn', 'shellcmd' or some other similar action, and then letting the program read from the named pipe or a flat file. The pipe or the file could then be updated with the 'write' action. The problem with the previous implementation of 'write' was the opening and closing the file at each write operation. A number of UNIX tools which can read from named pipes terminate if they see the writer closing the file. Therefore, you need to restart the program constantly (like by having a loop around the command line) which is not very efficient. In the most recent beta version, the 'write' action keeps the files permanently open which both speeds up writing, and also makes connecting external programs easier. In addition, there are now also actions to write to UNIX file system sockets and TCP/UDP sockets. So if the external program is able to listen on some socket and consume received data (e.g., like netcat does), the integration should again be relatively easy. kind regards, risto
On 02/21/2013 03:42 PM, Eric V. Smith wrote: > Sorry for the late reply. > > When I originally mentioned amqp it was to support the idea that a > persistent connection to a single process would solve most use cases. > And further, there's no need to build in to sec any sort of load > balancing across multiple instances of such connections. Instead, the > external program to which you have a persistent connection could use > something like amqp to do load balancing, queuing, etc. > > So, to be clear: I love the idea of a persistent connection to an > external program, and I don't think sec needs to do anything more > complicated than that. > > Eric. > > On 2/12/2013 11:44 AM, Boyles, Gary P wrote: >> I would think your original idea would help solve the issues... >> >>>> Therefore, would there be any interest to an output action which would >>>> leave the output open/running/connected, so that the next access of the >>>> same output would not have to do reopen/restart/connect again? This >>>> action would have to support a number of output types, e.g., >> >> -----Original Message----- >> From: Risto Vaarandi [mailto:risto.vaara...@seb.ee] >> Sent: Tuesday, February 12, 2013 8:26 AM >> To: Busko, Steve >> Cc: Simple-evcorr-users >> Subject: Re: [Simple-evcorr-users] extending output types (RabbitMQ) >> >> hi Steve, >> unfortunately, I am not too familiar with the RabbitMQ protocol as a >> developer, thus it is hard for me to tell how complex it would be to >> implement in the sec code. Nevertheless, my strong feeling is that it is >> much better idea to have it implemented not as a modification to the >> core code, but rather as a separate Perl module which can be loaded at >> sec startup. The number of potential application layer protocols is >> unfortunately vast, and providing active support and updates to many of >> them is a pretty hard task. On the other hand, since using code from >> external modules is quite easy with sec, I feel that support for >> specific protocols is best implemented in custom perl modules (also, >> there could be something already written at CPAN). >> However, since you have already apparently developed some code for >> RabbitMQ, why not to release it as a separate module to the public >> domain? There is a sec rule repository at the tool's home page which can >> be used for hosting this code. Also, if you separate your additions from >> core code into a module, upgrading sec will become much more easy. >> kind regards, >> risto >> >> On 02/11/2013 10:12 PM, Busko, Steve wrote: >>> Hello Ristro, >>> >>> While I don't have specific implementation feedback on this topic, I did >>> want to throw in that, like Gary, we are also using RabbitMQ and would like >>> to see sec integrate in some way possibly affecting this thread in part. >>> >>> Initially we used pipes, but found them not robust enough. So we tried >>> RabbitMQ and have been much happier with the flexibility and control. We >>> have been running it for around 6 months in a development environment >>> taking in around low 40s msgs/sec from a single syslog source and outputs >>> slightly less than half that into a single DB. In our case we didn't do >>> anything fancy to integrate, just inserted the MQ perl code/functionality >>> directly into sec and created an additional "queue" command to use vs. a >>> write. >>> >>> So we would like to support both the use of RabbitMQ and the consideration >>> of integrating it (or as a plug-in) with sec. >>> >>> Regards, >>> Steve Busko | NOC Tools Manager >>> >>>> -----Original Message----- >>>> From: Risto Vaarandi [mailto:risto.vaara...@seb.ee] >>>> Sent: Wednesday, February 06, 2013 12:38 PM >>>> To: Simple-evcorr-users >>>> Subject: [Simple-evcorr-users] extending output types >>>> >>>> hi all, >>>> >>>> as you know, currently there are number of outputs which are not >>>> directly supported by sec. For example, with 'pipe' action one can feed >>>> only one event to an external program (with most programs exiting after >>>> the 'pipe' action closes the pipe). Also, the 'write' action opens a >>>> given file before each access and closes it afterwards. While this makes >>>> output file rotation worry-free, it is not very efficient. Finally, sec >>>> does not directly support various sockets as output (for example, it is >>>> not possible to connect directly to a syslog server on another host and >>>> issue syslog events without any intermediaries). >>>> >>>> Therefore, would there be any interest to an output action which would >>>> leave the output open/running/connected, so that the next access of the >>>> same output would not have to do reopen/restart/connect again? This >>>> action would have to support a number of output types, e.g., >>>> >>>> write2 udp myserver:514 This is my event >>>> write2 tcp myserver:514 Another event >>>> write2 file /var/log/myevents Third event >>>> write2 program /bin/logger -p user.notice Fourth event >>>> >>>> It is probably easy to see that with sockets and files it is not that >>>> hard to distinguish them from each other, since the filename or server >>>> name plus port number can serve as unique identifiers. With command >>>> lines, however, it is somewhat more difficult, since an extra space or >>>> different ordering of options makes two command lines different, even if >>>> their effect is exactly the same. Also, since sec uses positional >>>> parameters, there is an issue with parsing, although this could be >>>> solved with the use of (). In addition, one workaround for both issues >>>> would be storing command line in an action list variable. >>>> The action outlines above would also need an output rotation mechanism, >>>> but this could be implemented with adding this to SIGUSR2 handler >>>> (closing all currently open outputs). >>>> >>>> Would the action above be of interest, and are there any other ideas for >>>> handling these situations? >>>> >>>> kind regards, >>>> risto >>>> >>>> ------------------------------------------------------------------------------ >>>> Free Next-Gen Firewall Hardware Offer >>>> Buy your Sophos next-gen firewall before the end March 2013 >>>> and get the hardware for free! Learn more. >>>> http://p.sf.net/sfu/sophos-d2d-feb >>>> _______________________________________________ >>>> Simple-evcorr-users mailing list >>>> Simple-evcorr-users@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >>> >>> >> >> >> ------------------------------------------------------------------------------ >> Free Next-Gen Firewall Hardware Offer >> Buy your Sophos next-gen firewall before the end March 2013 >> and get the hardware for free! Learn more. >> http://p.sf.net/sfu/sophos-d2d-feb >> _______________________________________________ >> Simple-evcorr-users mailing list >> Simple-evcorr-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >> >> ------------------------------------------------------------------------------ >> Free Next-Gen Firewall Hardware Offer >> Buy your Sophos next-gen firewall before the end March 2013 >> and get the hardware for free! Learn more. >> http://p.sf.net/sfu/sophos-d2d-feb >> _______________________________________________ >> Simple-evcorr-users mailing list >> Simple-evcorr-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >> > > ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
