We have a 3 node m1.large ES/logstash cluster with version 0.90.3. Everything has been running fine for a very long time.
We needed to upgrade our cluster to use m2.xlarge, so we killed a node and brought up a new brand new m2.xlarge machine and let the cluster go from yellow to green. But when we did the next machine, things went south. The latest logstash index has all of its primary and replica shards unassigned, and they are stuck that way. I'm attaching a screenshot of the head plugin of the last index (you can see that we have replica = 1): <https://lh3.googleusercontent.com/-pgfCiEKaBkE/Uz4cX0ad0yI/AAAAAAAADKY/BcRNvxa1sIY/s3200/Screen+Shot+2014-04-03+at+10.41.30+PM.png> The only exception i see in the logs is: [2014-04-04 01:59:06,829][DEBUG][action.admin.indices.close] [Crimson Dynamo] failed to close indices [logstash-2014.04.04] org.elasticsearch.indices.IndexPrimaryShardNotAllocatedException: [logstash-2014.04.04] primary not allocated post api| at org.elasticsearch.cluster.metadata.MetaDataIndexStateService$1.execute(MetaDataIndexStateService.java:95) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:285) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:143) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) I've tried restarting the cluster, but no love. My cluster is no longer consuming any logs because of this. Any idea how I can even begin to trouble shoot this? Thanks, alex -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8dc86c75-7a31-4885-8450-d44cc833140b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
