Hello, We have a 15 node cluster and have been experiencing intermittent cluster wide performance issues that last for between 1 and 5 minutes.
ES 1.1.0 jdk1.7.0_51 We started seeing this after upgrading from .0.9.3 to .1.1.0, although we have also made many other changes around the same time. It seems to happen about every other day. We start logging slow queries (for queries that normally run 5ms to 20ms) with times in the several hundred ms range for 10 to 15 seconds on all nodes, then the performance drops to several seconds. Once this happens our open contexts climb from in the hundreds to over 300K. At this point all of our queues fill up and we log many "org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000)". This usually lasts for 2-3 minutes and then everything recovers. Our clients time-out after 5 seconds and retry a couple of times contributing to the rapid queue build up. This has been happening during our lowest query rate times, it has not occurred during our busiest times (which are many times greater query rate). We have marvel installed and several other OS performance monitoring tools. We do see a small CPU spike when this happens (from almost nothing to 10% on 12 cores), but no swapping or high GC activity. Looking through all of the marvel graphs, other than open contexts going through the roof and average query latency climbing, everything else looks normal and really low utilization compared to our busy times. No large merging, no increase in indexing activity, etc. We have been through our application logs looking for for unusual query activity and I have captured wire data from every server on port 9200 during an occurrence and combed through the incoming search activity and have not been able to find anything to correlate to the cluster behaviour. Any advice on methods to troubleshoot this appreciated! Thanks, Jeff -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d3f22167-7404-4db0-9231-513bf7cf2108%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
