Among all features, I'd like to tackle this "catch and solve insert error" issue probably as the last one (there already is a lot of support for handling output errors, that's the main reason).
Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Vlad Grigorescu > Sent: Tuesday, April 10, 2012 4:55 PM > To: rsyslog-users > Subject: Re: [rsyslog] Who is interested in ElasticSearch? > > The thing to consider here is what happens when you have multiple > rsyslog servers logging to ElasticSearch. Does there need to be some > kind of concurrency, so that each of them have unique IDs for the > messages? What happens if two messages have the same ID? > > These are questions I'm unsure of, but for now, I'm happy to use > ElasticSearch's automatic ID generation features. > > --Vlad > > On 04/10/2012 09:49 AM, Radu Gheorghe wrote: > > 2012/4/10 <[email protected]>: > >> On Tue, 10 Apr 2012, Vlad Grigorescu wrote: > >> > >>> a) Messages that didn't get successfully inserted should probably > be > >>> queued and reattempted once or twice before being discarded. > Unfortunately, > >>> the new transactional interface won't be sufficient here - if > messages 1, 2, > >>> 4, and 5 are successfully inserted, but message 3 fails, as far as > I know, > >>> there's no way in the transactional interface to communicate that > only > >>> message 3 failed, instead of message 3-5. > >> > >> > >> actually, what happens is that rsyslog sends a transaction and gets > a single > >> success or failure message. > >> > >> if success, all messages were inserted > >> > >> if failure, it tries again with half as many messages to see if that > goes > >> through. If it gets down to one message and that fails, then it > considers it > >> a failure (and either retries, or drops the failed message) > >> > >> so if elasticsearch doesn't have transactions (all or none succeed), > then > >> some messages will be inserted multiple times. > > > > Maybe a solution to this is to use IDs somehow to avoid entering > > duplicates. Trying to add the same bulk (with the same IDs) will only > > "update" existing documents, and increment the "_version" number. > > > > I'm not sure how this could actually be implemented, but it might be > an option. > > > > BTW, I'm also interested in Elasticsearch :). But since I'm using it > > for logs, I'm not so much affected by duplicates. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > -- > Vlad Grigorescu | IT Security Engineer > Office of Privacy and Information Assurance > University of Illinois at Urbana-Champaign > 0x632E5272 | 217.244.1922 > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/
