Adrian, I ran the following command:
curl -XPUT http://localhost:9200/_settings -d '{"indices.store.throttle.max_bytes_per_sec" : "10gb"}' and received a { "acknowledged" : "true" } response. The logs showed "cluster state updated". I did have to close my index prior to changing the setting and reopen afterward. I've since began another optimize, but again it doesn't look like much is happening. The optimize isn't returning and the total CPU usage on every node is holding at about 2% of a single core. I would copy a hot_threads stack trace, but I'm unfortunately on a closed network and this isn't possible. I can tell you that refreshes of hot_threads show vary little happening. The occasional [merge] thread (always in a LinkedTransferQueue.awaitMatch() state) or [optimize] (doing nothing on a waitForMerge() call) thread shows up, but it's always consuming 0-1% CPU. It sure feels like something isn't right. Any thoughts? On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <[email protected] > wrote: > Did you see a message in the logs confirming that the setting has been > updated? It would be interesting to see the output of hot threads[1] to see > what your node is doing. > > [1] > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <[email protected]>wrote: > >> Yes. I have run max_num_segments=1 every time. >> >> >> On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick < >> [email protected]> wrote: >> >>> Have you tried max_num_segments=1 on your optimize? >>> >>> On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw >>> <[email protected]>wrote: >>> >>>> Any thoughts on this? I've run optimize several more times, and the >>>> number of segments falls each time, but I'm still over 1000 segments per >>>> shard. Has anyone else run into something similar? >>>> >>>> >>>> On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: >>>>> >>>>> OK. Optimize finally returned, so I suppose something was happening >>>>> in the background, but I'm still seeing over 6500 segments. Even after >>>>> setting max_num_segments=5. Does this seem right? Queries are a little >>>>> faster (350-400ms) but still not great. Bigdesk is still showing a fair >>>>> amount of file IO. >>>>> >>>>> On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node >>>>>> cluster, each with 64G of ram, with 24G allocated to Elasticsearch on >>>>>> each. I've batch loaded approximately 86 million documents into a single >>>>>> index (4 shards) and have started benchmarking cross_field/multi_match >>>>>> queries on them. The index has one replica and takes up a total of 111G. >>>>>> I've run several batches of warming queries, but queries are not as fast >>>>>> as >>>>>> I had hoped, approximately 400-500ms each. Given that *top *(on >>>>>> Centos) shows 5-8 GB of free memory on each server, I would assume that >>>>>> the >>>>>> entire index has been paged into memory (I had worried about disk >>>>>> performance previously, as we are working in a virtualized environment). >>>>>> >>>>>> A stats query on the index in questions shows that the index is >>>>>> composed of > 7000 segments. This seemed high to me, but maybe it's >>>>>> appropriate. Regardless, I dispatched an optimize command, but I am not >>>>>> seeing any progress and the command has not returned. Current merges >>>>>> remains at zero, and the segment count is not changing. Checking out hot >>>>>> threads in ElasticHQ, I initially saw an optimize call in the stack that >>>>>> was blocked on a waitForMerge call. This however has disappeared, and >>>>>> I'm >>>>>> seeing no evidence that the optimize is occuring. >>>>>> >>>>>> Does any of this seem out of the norm or unusual? Has anyone else >>>>>> had similar issues. This is the second time I have tried to optimize an >>>>>> index since upgrading. I've gotten the same result both time. >>>>>> >>>>>> Thanks in advance for any help/tips! >>>>>> >>>>>> - Elliott >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Adrien Grand > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFv00f0zXQ-pd77n9K5%3DqCE4rtp%3DXV2HLqREKsF%2Bf8hbXg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
