-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 4/11/12 2:33 AM, Radu Gheorghe wrote: > 2012/4/10 Vlad Grigorescu <[email protected]>: >> The thing to consider here is what happens when you have multiple rsyslog >> servers logging to ElasticSearch. Does there need to be some kind of >> concurrency, so that each of them have unique IDs for the messages? What >> happens if two messages have the same ID? > > If two messages have the same ID, the one that gets inserted last > overrides the previous one, and gets an incremented _version. Which > basically means you lose data, because the old message isn't there > anymore.
Well, that's certainly not what you want when it comes to logs. >> These are questions I'm unsure of, but for now, I'm happy to use >> ElasticSearch's automatic ID generation features. > > Well, if you rely on Elasticsearch to generate the IDs, I don't think > there's a way for rsyslog to know which documents were successfully > inserted and which not: > > The only way to know which document was inserted and which not is by > order. Which looks a bit risky in my book. According to the ES documentation[1] the order of the responses is the same as the order you sent to be indexed. I'll try to confirm with the author that that will remain the same down the line, but I suspect that many people rely on that fact at this point. --Vlad [1] - https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/bulk/BulkResponse.java#L32 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) iQIcBAEBCgAGBQJPhYARAAoJEMEVj6tjLlJyETsP/1SEB9lhS9JjeqvQTAgq6o1W W7Pgu22FxbWj+5VS3AkoDaBSapagrdc02YuWOrH+obMQ0smQExiCqrMq7JHpaI6G RVh0plhkwJDbJ2JmISQAED6ImjgRDSqBr9wHJ6Eoytm2SFmBaIzp9oNiCn5B6O/H CRh4PqL1KJcM99rvNv2MULkM8D0JRdeD+g3kXLhwm/0nUsI5Cw0zkxFcS1cSrXiL ODKCDgt0Wr4PnpmFAwqGizB2r8FFXTyoxHcYKWCQsCriyCeGe5ow2JLUYu2i/pqR GelC2SJVRuDdaiuTxV4UdXu8jcFrABxJt8UclOu2Bolq3t9Q2YZitVKzfteoFuKH OTgcHvQt9Qmns7Ew48cQHq+11oh7F6YG5KM9jtdeAjRyyQerLy8RQhL82T1ljHVl 45rLbgO+3BOCz9nuDlXm7jvjFUq0GO6zar/9TsYxEjNYJQZd0QrjcjSE1kCSMTsH 7eXwyVM9jxonaBzYufQQ6VuvSPznKqwvZe5phIYjmEFJnsL9tsritHATCctQykLm D5evrdRQq/iZ8Lpvkak1xEQo0Mb2xRJIn1MA1a62gkCWih5LVxf9yMwZhyj51xfc Ki/ss+BwrLdN7PI+eSouVMvN1s8y/kj0yEcoQBdR++0QlhzM3Vh3h9e6xruqI7B2 ZwEmR7NkZyfe6Y35k4yY =XE/L -----END PGP SIGNATURE----- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/
