Hello Ralph! I'm the initial author of the omhiredis output. I haven't looked at the omkafka source code yet so I can't speak to it's complexity - but I can say that writing the transaction support for omhiredis was fairly simple. It was just implementing the beginTransaction / endTransaction macro blocks in addition to the doAction block.
If knowing at the rsyslog level what is going on with your queues is important, I'd lean towards taking a crack at transaction support in omkafka. I'm currently using omelasticsearch heavily - and being able to look in my imstats logs to see what's happening with my queues and actions definitely makes it simple to tell what is going on at a glance. Just my thoughts! Cheers, Brian On Fri, Jan 15, 2016 at 12:31 AM, Ralph Caraveo <[email protected]> wrote: > Hello, > > (I apologize if this got sent twice but I may have sent it too soon before > the mailing list registration process was completed.) > > I'm hoping the group can provide some guidance around a requirement we > have to have transactional support around having an Rsyslog OM module that > writes to Kafka. > > What we'd like to do, is leverage consuming data from Rsyslog and posting > to Kafka however it looks like the OMKafka module doesn't currently support > transactions when posting to Kafka in the event of an error. So, if we > write to Kafka, and an error occurs, it looks like we lose the log-line for > that particular Action item unless we write it to a fallback log file. > > Additionally, we've looked at writing our own Kafka Producer using the > OMProg style where we consume off of STDIN and then connect to kafka and > produce data. Unfortunately this approach also doesn't allow us to > communicate back to Rsyslog that a failure has happened in the event of an > error. > > We tried to deal with this by writing to a fallback text file, and this > works great when there are errors with Kafka, but if the process dies > between receiving from STDIN and before writing to Kafka, we can still > potentially lose messages. > > Additionally, I noticed that the OMHIREDIS client does support > transactions, so it sounds like we want the design of OMHIREDIS (where it > utilizes transactions) but with that support in OMKafka. > > I'm just looking for a recommendation on a way forward from the group. If > it makes more sense to enhance OMKafka to have transaction support or if we > try to go down the path of adding some type of transaction support to the > Omprog module which would allow us to continue using our custom Kafka > producer of which is written in Go. > > Thanks for anyone's time around this! > > Ralph > @deckarep > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
