We have a cluster running 1.0.0 in Azure using unicast discovery. Recently we started seeing exceptions like these in the logs:
[2014-04-01 21:40:22,720][DEBUG][action.admin.indices.status] [ES2PROD-M01] [usg-2014-03-04][4], node[3cCeFKJrTMWaIhE3R6tlZA], [P], s[STARTED]: Failed to execute [ org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@2c06e67<https://github.com/org.elasticsearch.action.admin.indices.status.IndicesStatusRequest/elasticsearch/commit/2c06e675> ] org.elasticsearch.transport.NodeDisconnectedException: [ES2PROD-D07][inet[/10.0.64.68:9300]][indices/status/s] disconnected In this case D07 is still up and running. After several dozen of these exceptions, D07 is disconnected: [2014-04-01 21:40:24,096][INFO ][cluster.service ] [ES2PROD-M01] removed {[ES2PROD-D07][3cCeFKJrTMWaIhE3R6tlZA][es2prod-d07][inet[/10.0.64.68:9300]]{master=false},}, reason: zen-disco-node_failed([ES2PROD-D07][3cCeFKJrTMWaIhE3R6tlZA][es2prod-d07][inet[/10.0.64.68:9300]]{master=false}), reason transport disconnected (with verified connect) Four seconds later the same node is added back: [2014-04-01 21:40:28,712][INFO ][cluster.service ] [ES2PROD-M01] added {[ES2PROD-D07][3cCeFKJrTMWaIhE3R6tlZA][es2prod-d07][inet[/10.0.64.68:9300]]{master=false},}, reason: zen-disco-receive(join from node[[ES2PROD-D07][3cCeFKJrTMWaIhE3R6tlZA][es2prod-d07][inet[/10.0.64.68:9300]]{master=false}]) In the mean time the cluster goes yellow and starts recovery. This does not seem like a timeout type of issue since it happens so quickly, and then the disconnected node is added right back. Any ideas how we can get more info on the root cause and avoid this from happening? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/df768c5e-5833-42b5-a804-b7d07f51996b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
