2012/4/10 <[email protected]>: > On Tue, 10 Apr 2012, Vlad Grigorescu wrote: > >> a) Messages that didn't get successfully inserted should probably be >> queued and reattempted once or twice before being discarded. Unfortunately, >> the new transactional interface won't be sufficient here - if messages 1, 2, >> 4, and 5 are successfully inserted, but message 3 fails, as far as I know, >> there's no way in the transactional interface to communicate that only >> message 3 failed, instead of message 3-5. > > > actually, what happens is that rsyslog sends a transaction and gets a single > success or failure message. > > if success, all messages were inserted > > if failure, it tries again with half as many messages to see if that goes > through. If it gets down to one message and that fails, then it considers it > a failure (and either retries, or drops the failed message) > > so if elasticsearch doesn't have transactions (all or none succeed), then > some messages will be inserted multiple times.
Maybe a solution to this is to use IDs somehow to avoid entering duplicates. Trying to add the same bulk (with the same IDs) will only "update" existing documents, and increment the "_version" number. I'm not sure how this could actually be implemented, but it might be an option. BTW, I'm also interested in Elasticsearch :). But since I'm using it for logs, I'm not so much affected by duplicates. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/
