Hey All, We're on ES 1.4.3. Initially, I thought I had this issue, however it appears I was incorrect.
https://github.com/elasticsearch/elasticsearch/issues/9023 We're seeing very uneven primary shard distribution as we're adding new indexes to our system. We're running a 6 node cluster. We're adding indexes with 12 shards, and 1 replica. What I'm noticing is that 1/4 (3) of the primary shards are be allocated to a single node. In the case of our application, this is causing severe performance problems since we're writing 1/3 of our throughput to a single node, leaving the remaining 2/3 spread over 5 nodes. We currently have 87 indexes and growing. I never noticed this with a handful of indexes, but as we're getting more it's becoming more prevalent. What can I do to diagnose and fix this issue? From the screenshot above, I've tried moving the primary shards, but this seems to have little effect. When catting the shards and checking how many primaries exist per node, they're very skewed curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res001sy|wc -l => 259 curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res002sy|wc -l => 228 curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res003sy|wc -l => 191 curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res004sy|wc -l => 106 curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res005sy|wc -l => 37 curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res006sy|wc -l => 13 As you can see, 01 is our highest with 259 primary shards, however 06 only had 13. I've tried forcing shards to be rebalance with the command below but it seems to have no effect. Also, note that we're running the aws cloud plugin. https://github.com/elasticsearch/elasticsearch-cloud-aws. curl -XPOST 'res002sy:9200/_cluster/reroute' -d '{}' Any ideas what I can do to resolve this problem? The uneven distribution seems to killing our more heavily nodes.since they're taking a majority of the primary shard throughput. I'd prefer that most nodes have an approximately equal amount of primaries and secondaries, since our indexes are mixes of more write heavy or more read heavy. We seem to have very few that are 50/50. Thanks, Todd -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7c935128-e783-48cc-8134-3bc2de0c9aaa%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
