Hi Adrien,
I did customize my merge policy, although I did so only because I was so
surprised by the number of segments left over after the load. I'm pretty
sure the optimize problem was happening before I made this change, but
either way here are my settings:
"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}
Not sure whether this set up could be a contributing factor or not.
Nothing really jumps out at me in the logs. In fact, when i kick off the
optimize, I don't see any logging at all. Should I?
I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize
Thanks!
On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:
>
> Hi Elliott,
>
> 1500 segments per shard is certainly way too much, and it is not normal
> that optimize doesn't manage to reduce the number of segments.
> - Is there anything suspicious in the logs?
> - Have you customized the merge policy or scheduler?[1]
> - Does the issue still reproduce if you restart your cluster?
>
> [1]
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html
>
>
>
> On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw
> <[email protected]<javascript:>
> > wrote:
>
>> Any other thoughts on this? Would 1500 segments per shard be
>> significantly impacting performance? Have you guys noticed this behavior
>> elsewhere?
>>
>> Thanks.
>>
>>
>> On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote:
>>>
>>> Adrian,
>>>
>>> I ran the following command:
>>>
>>> curl -XPUT http://localhost:9200/_settings -d
>>> '{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'
>>>
>>> and received a { "acknowledged" : "true" } response. The logs showed
>>> "cluster state updated".
>>>
>>> I did have to close my index prior to changing the setting and reopen
>>> afterward.
>>>
>>>
>>> I've since began another optimize, but again it doesn't look like much
>>> is happening. The optimize isn't returning and the total CPU usage on
>>> every node is holding at about 2% of a single core. I would copy a
>>> hot_threads stack trace, but I'm unfortunately on a closed network and this
>>> isn't possible. I can tell you that refreshes of hot_threads show vary
>>> little happening. The occasional [merge] thread (always in a
>>> LinkedTransferQueue.awaitMatch() state) or [optimize] (doing nothing on
>>> a waitForMerge() call) thread shows up, but it's always consuming 0-1%
>>> CPU. It sure feels like something isn't right. Any thoughts?
>>>
>>>
>>>
>>>
>>> On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <adrien...@elasticsearch.
>>> com <javascript:>> wrote:
>>>
>>>> Did you see a message in the logs confirming that the setting has been
>>>> updated? It would be interesting to see the output of hot threads[1] to
>>>> see
>>>> what your node is doing.
>>>>
>>>> [1] http://www.elasticsearch.org/guide/en/elasticsearch/
>>>> reference/current/cluster-nodes-hot-threads.html
>>>>
>>>>
>>>> On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw
>>>> <[email protected]<javascript:>
>>>> > wrote:
>>>>
>>>>> Yes. I have run max_num_segments=1 every time.
>>>>>
>>>>>
>>>>> On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
>>>>> [email protected] <javascript:>> wrote:
>>>>>
>>>>>> Have you tried max_num_segments=1 on your optimize?
>>>>>>
>>>>>> On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw
>>>>>> <[email protected]<javascript:>
>>>>>> > wrote:
>>>>>>
>>>>>>> Any thoughts on this? I've run optimize several more times, and the
>>>>>>> number of segments falls each time, but I'm still over 1000 segments
>>>>>>> per
>>>>>>> shard. Has anyone else run into something similar?
>>>>>>>
>>>>>>>
>>>>>>> On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote:
>>>>>>>>
>>>>>>>> OK. Optimize finally returned, so I suppose something was
>>>>>>>> happening in the background, but I'm still seeing over 6500 segments.
>>>>>>>> Even
>>>>>>>> after setting max_num_segments=5. Does this seem right? Queries are
>>>>>>>> a
>>>>>>>> little faster (350-400ms) but still not great. Bigdesk is still
>>>>>>>> showing a
>>>>>>>> fair amount of file IO.
>>>>>>>>
>>>>>>>> On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote:
>>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node
>>>>>>>>> cluster, each with 64G of ram, with 24G allocated to Elasticsearch on
>>>>>>>>> each. I've batch loaded approximately 86 million documents into a
>>>>>>>>> single
>>>>>>>>> index (4 shards) and have started benchmarking
>>>>>>>>> cross_field/multi_match
>>>>>>>>> queries on them. The index has one replica and takes up a total of
>>>>>>>>> 111G.
>>>>>>>>> I've run several batches of warming queries, but queries are not as
>>>>>>>>> fast as
>>>>>>>>> I had hoped, approximately 400-500ms each. Given that *top *(on
>>>>>>>>> Centos) shows 5-8 GB of free memory on each server, I would assume
>>>>>>>>> that the
>>>>>>>>> entire index has been paged into memory (I had worried about disk
>>>>>>>>> performance previously, as we are working in a virtualized
>>>>>>>>> environment).
>>>>>>>>>
>>>>>>>>> A stats query on the index in questions shows that the index is
>>>>>>>>> composed of > 7000 segments. This seemed high to me, but maybe it's
>>>>>>>>> appropriate. Regardless, I dispatched an optimize command, but I am
>>>>>>>>> not
>>>>>>>>> seeing any progress and the command has not returned. Current merges
>>>>>>>>> remains at zero, and the segment count is not changing. Checking out
>>>>>>>>> hot
>>>>>>>>> threads in ElasticHQ, I initially saw an optimize call in the stack
>>>>>>>>> that
>>>>>>>>> was blocked on a waitForMerge call. This however has disappeared,
>>>>>>>>> and I'm
>>>>>>>>> seeing no evidence that the optimize is occuring.
>>>>>>>>>
>>>>>>>>> Does any of this seem out of the norm or unusual? Has anyone else
>>>>>>>>> had similar issues. This is the second time I have tried to optimize
>>>>>>>>> an
>>>>>>>>> index since upgrading. I've gotten the same result both time.
>>>>>>>>>
>>>>>>>>> Thanks in advance for any help/tips!
>>>>>>>>>
>>>>>>>>> - Elliott
>>>>>>>>>
>>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "elasticsearch" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected] <javascript:>.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/elasticsearch/5391291f-
>>>>>>> 5c5e-4088-a1f2-93272beef0bb%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to a topic in
>>>>>> the Google Groups "elasticsearch" group.
>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/
>>>>>> topic/elasticsearch/kqTRRADQBwc/unsubscribe.
>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>> [email protected] <javascript:>.
>>>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>>>> msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%
>>>>>> 3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "elasticsearch" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected] <javascript:>.
>>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>>> msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%
>>>>> 3Demc0JTouT9%2BBeUk_A%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Adrien Grand
>>>>
>>>> --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "elasticsearch" group.
>>>> To unsubscribe from this topic, visit https://groups.google.com/d/
>>>> topic/elasticsearch/kqTRRADQBwc/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> [email protected] <javascript:>.
>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>> msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%
>>>> 2BKDsg%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Adrien Grand
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/775e2e32-5891-4514-bd53-32b52c0deb1a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
