Hey all, Sorry to resurrect a dead thread. Did you ever find a solution for eventual consistency of documents across EC2 regions?
Thanks, todd On Wednesday, May 1, 2013 5:50:00 AM UTC-7, Norberto Meijome wrote: > > +1 on all of the above. es-reindex already in my list of things to > investigate (for a number of issues...) > > cheers, > b > > > On Wed, May 1, 2013 at 6:58 AM, Paul Hill <[email protected] <javascript:> > > wrote: > >> On 4/23/2013 8:44 AM, Daniel Maher wrote: >> >>> On 2013-04-23 5:22 PM, Saikat Kanjilal wrote: >>> >>>> Hello Folks, >>>> [...] does ES out of the box currently support cross data >>>> center replication, [....] >>>> >>> >>> Hello, >>> >>> I'd wager that the question you're really asking about is how to control >>> where shards are placed; if you can make deterministic statements about >>> where shards are, then you can create your own "rack-aware" or "data >>> centre-aware" scenarios. ES has supported this "out of the box" for well >>> over a year now (possibly longer). >>> >>> You'll want to investigate "zones" and "routing allocation", which are >>> the key elements of shard placement. There is an excellent blog post which >>> describes exactly how to set things up here : >>> http://blog.sematext.com/2012/05/29/elasticsearch-shard- >>> placement-control/ >>> >>> Is shard allocation really the correct solution if the data centers are >> globally distributed? >> >> If I have a data center in the US intended to server data from the US, >> but it should also have access to Europe and Asia data, and clusters in >> both Europe and Asia with similar needs, would I really want to use zones >> etc. and have one great global cluster with data center aware >> configurations? >> >> Assuming that the US would be happy to deal with old documents from Asia >> and Europe, when Asia or Europe is off line or just not caught up, it would >> seem that you would NOT want a "world" cluster, because I can't picture how >> you'd configure a 3-part world cluster for both index into the right >> indices, search the right (possible combination of) shards, but also >> preventing "split brain". >> >> In the scenerio, I've described, I would think each data center might >> better provide availability and eventual consistency (with less concern for >> the remote data from the other region) by having three clusters and some >> type of syncing from one index to copies at the other two locations. For >> example, the US datacenter might have a US, copyOfEurope, and copyOfAsia >> index. >> >> Anyone have any observations about such a world-wide scenerio? >> Are there any index to index copy utilities? >> Is there a river or other plugin that might be useful for this three >> clusters working together scenerio? >> How about the project https://github.com/karussell/elasticsearch-reindex? >> Comments? >> >> -Paul >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > > -- > Norberto 'Beto' Meijome > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/646067d1-1137-4777-be51-ced0bd6a3edd%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
