Note, if you want to connect a database to ES internals, this is generally a bad idea. The challenge is performance and transaction safety. ES API can process many hundreds of thousands of actions in a second in unspecified order, like a "data hose". No database exist that I know of that can keep up with such a pace. The price would be a slowdown of ES throughput (async also means a higher cost of CPU for threads and RAM for the message queue on the same machine which would compete heavily with ES for resources). >From the viewpoint of designing such an architecture, never do a "reverse river" for "every time a document is inserted or modified". Instead, just push the doc to ES and to the database concurrently from your middleware only if necessary, i.e. after applying a filter condition to your doc. This would also maintain scalability and transaction safety since there is no guarantee that ES transport layer does not drop (or repeat) transmissions at any time.
Beside these design questions, there is a new "mock transport" that has been added to 1.2.0. Nice side effect is that plugin authors should be able to completely exchange the transport module or wrap the existing one with a logging facility. I think about a syslog client for revisable logs that capture certain ES transport actions and write them into an append-only log on a central log server, so it's up to the syslog server to keep up with the pace. With a bit filtering it could be possible to capture certain documents and log them. If this can work out is still totally unclear, I'm still in the process of designing :) Jörg On Fri, May 23, 2014 at 5:45 AM, Tim Uckun <[email protected]> wrote: > I would like to have a river in reverse. Every time a document is inserted > or modified I would like to push that into another destination like a > database. Ideally this would be async or maybe even in batches. > > Has anybody done anything like this before? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/45b10d5a-1981-46a5-bb4a-a7ce9273ba7f%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGbNbxBV20XA0nF_bRQy%3D_g5zZjtLB1eYYm36aSZOsA1w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
