So it doesn't appear that the sum of currentqueries = K * active search threads . In other words they are not proportional.
Currentqueries was flat, and search.active jumped 2 orders of magnitude. We have highly varied indexes and searches. Some indexes have only one shard with one replica and some have 40 shards with 1 replica. Some queries are nothing more than a search string with no boosting ordered by a date and with no facets. Others have 6 facets, boosts, interesting analyzers and ordered by score. I'm sure some searches use more resources than others, but do they expand to multiple threads? I would imagine one could easily expand this out at the shard (or maybe segment) level. I could also imagine that facets could get their own threads once the reduced document set was determined. --Shannon Monasco P.S. If you don't know about version 0.19.3, but know about .90+ or even 1.x that's fine. I'll take answers I can get. On Friday, July 11, 2014 9:58:03 PM UTC-6, smonasco wrote: > > Thanks Ivan! > On Jul 11, 2014 5:56 PM, "Ivan Brusic" <[email protected]> wrote: > >> The default in 0.90 (not sure about 0.19.) should still be a fixed search >> thread pool: >> http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/modules-threadpool.html >> >> I find that the current queries and active threads tends to be the same >> number. When I look at my graphs, they have the same movements. >> >> If you do not have any queued requests you might want to set a lower cap >> if your cluster is experiencing slowness. >> >> BTW, the best course of action that you can take is to simply upgrade and >> not worry about thread settings. The Lucene 4 improvements (Elasticsearch >> 0.90 I believe) were monumental in the space savings. New versions will >> offer tons of other improvements, but the 0.90 release was especially great! >> >> Cheers, >> >> Ivan >> >> >> On Fri, Jul 11, 2014 at 3:09 PM, Shannon Monasco <[email protected]> >> wrote: >> >>> I have never seen any queuing on search threads. Not sure if in .19 it >>> defaults to cache or not but that's the behavior I see. >>> >>> What about current queries from indices stats? >>> On Jul 11, 2014 2:53 PM, "Ivan Brusic" <[email protected]> wrote: >>> >>>> Your second paragraph is correct. The threads are the total number of >>>> search threads at your disposal, active is the number of ongoing threads >>>> and queue are the number of threads that cannot be run since your thread >>>> pool is exhausted, which should be when active == threads, but not always >>>> the case. >>>> >>>> The default number of search threads is based upon the number of >>>> processors (3x # of available processors). There is no good metric for >>>> determining a balance since searches can be either lightweight >>>> (milliseconds) or heavyweight (minutes), but I would argue that the key >>>> metric to monitor is your queue. Is it normally empty? Spiky behavior? >>>> Requests constantly queued? >>>> >>>> >>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html >>>> >>>> Cheers, >>>> >>>> Ivan >>>> >>>> >>>> On Fri, Jul 11, 2014 at 1:19 PM, smonasco <[email protected]> wrote: >>>> >>>>> I should probably preface everything with I'm running a 5 node cluster >>>>> with version 0.19.3 and should be up to version 1.1.2 by the middle of >>>>> August, but I have some confusion around metrics I'm seeing, what they >>>>> mean >>>>> and what are good values. >>>>> >>>>> In thread_pools I see threads, active and queued. Queued + active != >>>>> threads. I assume this really is a work pool and you have active >>>>> threads, >>>>> a thread count in the work pool and queued work. So some explanation >>>>> around this would be nice. >>>>> >>>>> I've correlated some spikes in search threads with heap mem >>>>> utilization explosions. current searches sort of also correlate, but I >>>>> have more current searches than search threads and there is not search >>>>> threadpool queueing. >>>>> >>>>> I'm not sure how current searches correlate (or if they should/do) >>>>> with search threads. >>>>> >>>>> I've observed the following: >>>>> >>>>> Devestating: 10,000 current searches on worst index sustained over >>>>> hours with not much change ending at the same time as spikes of > 1000 >>>>> search threads (where we generally average < 50) and a heap explosion. >>>>> >>>>> Oddly OK: current searches averaging 15 on worst index spiking to 105 >>>>> with search threads averaging 50 with maxes of 300 spiking to averages of >>>>> 120 and maxes of > 1000 >>>>> >>>>> >>>>> So... I guess, what are good ranges for search threads and current >>>>> searches? >>>>> >>>>> --Shannon Monasco >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e4d4bf22-e57c-4520-9418-23d2c0e7b56e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
