We have built elastic search cluster for our application autosuggest and we are facing one strange issue, need immediate help. Below is the setup of ES cluster:
We built the azure elastic search cluster with: Master only Nodes: 3 medium VMs with 7 GB RAM Non-master non-data nodes: 3 medium VMs with 7 GB RAM Data nodes: 3 large VMs with 14 GB RAM ES version: 1.1.1 Java version: 1.7.0_45 Index size is: 1GB Replica:1 Shards: 5 Java Heap size is half of RAM size. In a definite pattern everyday, we see the number of requests increases x100. I think our cluster should be able to handle that load, but 99% of search requests goes only on one node(and everyday that node keeps changing), and other node have only 1% traffic. Because of this, the search thread pool queue increases on that node. If that load would have divided on all the data nodes then we wouldn't have had any problem. I have increased the search thread pool count to 2000 on each of data nodes. But when requests increases on one node all threads get occupied and queue start building. Today I mitigated it by setting the search thread pool type to cached (that will have unbounded search thread pool). Please let me know if anyone faced similar issue or know how to make the load balancer work properly. We are using bigdesk to monitor the traffic, please let me know if you need any more information. Thanks, Manish -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO4PsGGJkP5pwCypegbnS_ik3%3D8NVMe3b06f_3gZgQ%3DQHrpXag%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
