-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 First off, I'm very interested in ElasticSearch. I tried several different backend databases for log storage, and none of them could scale as well. With a single moderately-sized ES server, I was able to index over 3000 DNS query logs per second, and querying the data was very fast. I have a lot more data to index (~50k/s), and am currently building out the ES cluster.
On 4/10/12 6:10 AM, Radoslav Bodó wrote: > b) logstash has better functionality in parsing and mangling data before > they are pushed to els Logstash is easier to configure, yes. But in my experience, it was unstable and couldn't keep up with any significant amount of logs. I wasn't using any feature in logstash that rsyslog doesn't have - I was using it for message filtering and normalization (which it only does via regular expressions, which were slow). > c) els clients are very sensitive to input data. there were case when > there were binary data in logs and those cannt be pushed and whole > cluster crashed because of this. You can easily escape this in rsyslog, and configure the character used to escape it. Also, with the newer ES versions, I have yet to experience a crash in sending the data to ES. Here's my current wishlist for rsyslog/elasticsearch integration: 1) Support bulk inserts (<http://www.elasticsearch.org/guide/reference/api/bulk.html>). 2) Parse the reply, for two things: a) Messages that didn't get successfully inserted should probably be queued and reattempted once or twice before being discarded. Unfortunately, the new transactional interface won't be sufficient here - if messages 1, 2, 4, and 5 are successfully inserted, but message 3 fails, as far as I know, there's no way in the transactional interface to communicate that only message 3 failed, instead of message 3-5. b) Messages that matched a percolator should be processed differently. A percolator (<http://www.elasticsearch.org/guide/reference/api/percolate.html>) is a saved query on the ES cluster. Whenever a message is inserted that matches a percolator, it is indicated in the response {"matches": "system_failed"}. This provides near-realtime search functionality. Anything that matches a percolator should somehow be reentered into the queue, so it can be passed to another output plugin (out to a file, ommail, etc.) 3) The ES server and port should be configured via config directives. 4) Somehow, the index and type for each message should be passed to the elasticsearch plugin. This is a bit tricky, because if it's part of the message itself, it takes some time to parse that data out of the message. 5) ES has an automatic discovery feature, where it will detect other cluster members (<http://www.elasticsearch.org/guide/reference/modules/discovery/zen.html>). Ideally, rsyslog would also use this, so that if a cluster member goes down, it can find a new cluster member, and the system benefits from the high-availability of elasticsearch. We have a developer that's currently working on many of these features, so I'm happy to offer some assistance with building this out. --Vlad Grigorescu -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) iQIcBAEBCgAGBQJPhDATAAoJEMEVj6tjLlJyRi0QAJduzSmE/xZDOmpkRuDAFfm9 UNGeRQNJIUQJGlNS3+Auk+k7714KoQhGkjHiUqKb23QpPxTVEbOSCoRMdfwrrzq/ zQ9F58XdKbDd29/+0YBuO0m6l4CAqB8x6IlRnYjcWNdjLV8EjhXZrff8vV6MDOPc WZZZ/GRTbKHdhVPhfLJMCtmqau3hYdR7qTW8hIkMpwS8nL9JrHrhTY6+F3bPzjI7 YF3IGKed+raV/3/VgV+aoBucjRwk8A5TSo8DuXJqDOZHjxLsjZ8t2K9PdSvZPjY9 gG/eK8dCKdswgZM+tv9TkJurwV+NOFPEgfvcpehJowuY3UzfsRg/tzHWehn84pWg iBSUbWJ3J7f+4Q9ky3XARS/R0Ebx4Igs5DODqsI2SXg11DCg4Ll0D5fF12+ybZDh VE1n6vLLuPxE2z8rXq8Oj/SQVvyWJBEu/jA3ibtcLi07fsEHP/3bQNc3LHR/ZTc5 /thotJscKrKY5ETpIYxBRdd33bVN+NxydBAbgcDJl4dt41hs2s6WP+Fb7ilWDmOt H1i3CLeTiFoyEx/9EqRDvNpjed29tr4x8KXMUU9l1Zm+4Ul2rDJtpB8adUrvT7Jr tQ7kbewlfxvCME1wZm5BHglb1C034B5yRdZcesg57CGmueNtptnkq+e983ezg/Ln g12Bf2Uvx24X5t8W7grt =RCu3 -----END PGP SIGNATURE----- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/
