Finally found query causing claster crashes. After I commented code doing
this - claster is ok for few days. Before it it was crashing once day in
average.
Query looks like:
{
"sort": [
{
"user_last_contacted.ct": {
"nested_filter": {
"term": {
"user_last_contacted.owner_id":
"542b2b7fb0bc2244056fd90f"
}
},
"order": "desc",
"missing": "_last"
}
}
],
"query": {
"filtered": {
"filter": {
"term": {
"company_id": "52c0e0b7e0534664db9dfb9a"
}
},
"query": {
"match_all": {}
}
}
},
"explain": false,
"from": 0,
"size": 100
}
mapping looks like:
"contact": {
"_all": {
"type": "string",
"enabled": true,
"analyzer": "default_full",
"index": "analyzed"
},
"_routing": {
"path": "company_id",
"required": true
},
"_source": {
"enabled": false
},
"include_in_all": true,
"dynamic": false,
"properties": {
"user_last_contacted": {
"include_in_all": false,
"dynamic": false,
"type": "nested",
"properties": {
"ct": {
"include_in_all": false,
"index": "not_analyzed",
"type": "date"
},
"owner_id": {
"type": "string"
}
}
}...
This query causes elasticsearch to eat more and more heap and finally if
crashed with java.lang.OutOfMemoryError: Java heap space. Also it's clear
that ES doesn't clear heap after such queries. This is started to occur
after upgrade from ES 0.90
It could be a number of things. Check your various ES caches. Full?
> Correlated with GC activity increase and eventual OOM. Then check your
> queries - are they big? Expensive aggregations? (the other day I saw one of
> our clients using agg queries 10K lines in size) I could keep asking
> questions..... share everything you've got to get help here.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Thursday, November 20, 2014 3:53:24 AM UTC-5, tetlika wrote:
>>
>> anyone?
>>
>> Середа, 19 листопада 2014 р. 13:32:37 UTC+1 користувач Serg Fillipenko
>> написав:
>>>
>>> We have contact profiles (20+ fields, containing nested documents)
>>> indexed and their social profiles(10+ fields) indexed as child documents of
>>> contact profile.
>>> We run complex bool match queries, delete by query, delete children by
>>> query, faceting queries on contact profiles.
>>> index rate 14.31op/s
>>> remove by query rate 13.41op/s (such high value caused by fact we
>>> delete all child docs first before indexing of parent and then we index
>>> children again)
>>> search rate 2.53op/s
>>> remove by ids 0.15op/s
>>>
>>> We started to face this trouble under ES 1.2 but just after we started
>>> to index and delete (no searching requests yet) child documents. On ES 1.4
>>> we have the same issue.
>>>
>>>
>>> What sort of data is it, what sort of queries are you running and how
>>>> often are they run?
>>>>
>>>> On 19 November 2014 17:52, tetlika <[email protected]> wrote:
>>>>
>>>>> hi,
>>>>>
>>>>> we have 6 servers and 14 shards in cluster, the index size 26GB, we
>>>>> have 1 replica so total size is 52GB, and ES v1.4.0, java version
>>>>> "1.7.0_65"
>>>>>
>>>>> we use servers with RAM of 14GB (m3.xlarge), and heap is set to 7GB
>>>>>
>>>>> around week ago we started facing next issue:
>>>>>
>>>>> random cluster servers around once per day/two are hitting the heap
>>>>> size limit (java.lang.OutOfMemoryError: Java heap space) in log, and
>>>>> cluster is failing - becomes red or yellow
>>>>>
>>>>> we tried adding more servers to cluster - even 8, but than it's a
>>>>> matter of time when we'll hit the problem, so looks no matter how many
>>>>> servers are in cluster - it will still hit the limit after some time
>>>>>
>>>>> before we started facing the problem we were running smoothly with 3
>>>>> servers
>>>>> also we set indices.fielddata.cache.size: 40% but it didnt helped
>>>>>
>>>>> also, there are possible workarounds to decrease heap usage:
>>>>>
>>>>> 1) reboot some server - than heap becomes under 70% and for some time
>>>>> cluster is ok
>>>>>
>>>>> or
>>>>>
>>>>> 2) decrease number of replicas to 0, and than back to 1
>>>>>
>>>>> but I dont like to use those workarounds
>>>>>
>>>>> how it can happen while all index can fit into RAM it can run out of
>>>>> it?
>>>>>
>>>>> thanks much for possible help
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "elasticsearch" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/elasticsearch/2ae23017-fde7-4b10-b31b-39076b079f10%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/elasticsearch/2ae23017-fde7-4b10-b31b-39076b079f10%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6cd54c22-26f7-40c0-95a9-ab960496e0f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
