> 1) Support bulk inserts > (<http://www.elasticsearch.org/guide/reference/api/bulk.html>). > 2) Parse the reply, for two things: > a) Messages that didn't get successfully inserted should probably be > queued and reattempted once or twice before being discarded. > Unfortunately, the new transactional interface won't be sufficient here > - if messages 1, 2, 4, and 5 are successfully inserted, but message 3 > fails, as far as I know, there's no way in the transactional interface > to communicate that only message 3 failed, instead of message 3-5. > b) Messages that matched a percolator should be processed > differently. A percolator > (<http://www.elasticsearch.org/guide/reference/api/percolate.html>) is > a saved query on the ES cluster. Whenever a message is inserted that > matches a percolator, it is indicated in the response {"matches": > "system_failed"}. This provides near-realtime search functionality. > Anything that matches a percolator should somehow be reentered into the > queue, so it can be passed to another output plugin (out to a file, > ommail, etc.)
I need to dig into more details with this feature. It looks like we could use a kind of status indicator, just like recently added for the normalizer, for this functionality (As far as the rsyslog engine is concerned). > 3) The ES server and port should be configured via config directives. That's already available with the new version. Doc is missing, simply because things are not yet finalized Action(type="omelasticsearch" server="" port="" ...) > 4) Somehow, the index and type for each message should be passed to the > elasticsearch plugin. This is a bit tricky, because if it's part of the > message itself, it takes some time to parse that data out of the > message. It's currently a per-instance fixed (but configurable) string. I've already begun to discuss a functionality similar to dynafiles to permit message-contained values. > 5) ES has an automatic discovery feature, where it will detect other > cluster members > (<http://www.elasticsearch.org/guide/reference/modules/discovery/zen.ht > ml>). Ideally, rsyslog would also use this, so that if a cluster member > goes down, it can find a new cluster member, and the system benefits > from the high-availability of elasticsearch. > > We have a developer that's currently working on many of these features, > so I'm happy to offer some assistance with building this out. It would probably be quite good for all parties involved if we could join forces. I obviously can provide strong rsyslog knowledge and would love to work with someone who is far more fluent as I am in ES ;) Just reply (on- or off-list as you like) to sort out any questions. Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/
