Hi all! First, I'd like to thank the creators of ES, the contributors, and the large community that surrounds it. I've gotten all sorts of great information from this list, the IRC channel, and all across the web. But, onto my scenario/question...
Long story (question below): Yesterday, my task was to simulate a disaster scenario with my Elasticsearch cluster, just to see how easy it would be to pull through the mess. At the time, I was working with a single cluster, and three total nodes. Two nodes were active when I started, and one was a node that was completely fresh, didn't have the ES service running on it yet, and had never yet joined this (or any) cluster. All of the indexes I care about have shards with 1 replica. I started by ungracefully shutting down the first of the two nodes, and letting the second node take all of the activity. Everything was good so far. Obviously no data loss... onto step two. I brought the first node back up. I then wanted to simulate the scenario where the second node couldn't handle the extra work of rebalancing shards as well as handling most or all indexing/query traffic, so I ungracefully shut down elasticsearch on the second node as well. At this point, obviously, the cluster health was red, and a good portion of my data was missing. Feeling confident that I could still recover from this situation, I shut down elasticsearch on the first node, effectively shutting off the cluster completely across the board. Then I started ES on my third node (the fresh one), followed by starting ES on the first and second nodes as well. Much to my delight, all of the rebalancing took place, and all of my indexes and shards seemed to survive the storm just fine. I didn't get to learn any lessons about how to recover from that sort of disaster, because apparently it wasn't disastrous enough! HOWEVER... fast-forward to today... I have some daily indexes, and their mappings are set by a template that was created via the API (not via a config file), and today I noticed that the field mappings were wrong. I checked for the template via the API, and to my dismay, it was nowhere to be found. Question: Where/how are templates created via the rest API stored? Are they persisted to disk in any way, or are they just a bit of information that's shared amongst all of the nodes in a cluster, such that if all nodes were to be shut down at the same time, the template would be lost? How about in a split brain situation, which I introduced in my scenario, to an extent. If a new master comes in and doesn't have knowledge of any custom templates, but then another node -- which previously had those templates defined -- joins the cluster, does the fresh master win, therefore blowing away the template? I'd be more than happy to set up some more test scenarios, and try to reproduce this, if it would help... but if it's a known issue, or if it just works that way by design, then I would rather not spend the time. Thoughts? Thanks! Andy Walker -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4ff9593b-80a4-4eee-953d-08d287a7b683%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
