Kafka looks interesting, though at this point we're actively trying to reduce the number of moving parts, so I think an AMQ based approach is what we'll ultimately go for. Seems like there might be room here for an elasticsearch-elasticsearch-river plugin or something - to do one or two way close to real time replication on some selected set of indexes between separate clusters. That way you could easily mirror prod data to a dev environment without depending on the ability to do the duplication earlier in the pipeline, or depending on scripts to move the data around.
On Wednesday, March 12, 2014 2:46:19 PM UTC-7, Otis Gospodnetic wrote: > > Consider Kafka 0.8.1. It comes with a MirrorMaker tool that mirrors Kafka > data (to multiple DCs). Once data is local, you can feed your ES from the > local Kafka broker. > > Otis > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > > On Wednesday, March 12, 2014 2:55:58 PM UTC-4, Josh Harrison wrote: >> >> Say I have clusters A and B. Cluster A is consuming data using an >> ActiveMQ river. I would like to stream data to cluster B as well. Do I just >> create a secondary outbound AMQ channel and subscribe cluster B to it, or >> is there a decent way to have a live copy of data going two places at once? >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6b1bdbe4-e2fa-4b10-9298-62d3d1869842%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
