Thanks for the help.
I am getting errors like this, which i interpret as that execution of a
query requested more than available memory.
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.FixedBitSet.<init>(FixedBitSet.java:54)
at
org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:104)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:311)
...
What bothers me, is why our recent addition (last month) of two nodes (from
6 to 8) to the cluster has not solved this issue.
What i expect, is that growing the cluster from 6 to 8 nodes means each
node has about 25% more resources available. Less field data loaded per
node (since it now has less shards) and less data/shards to aggregate over.
I am looking for an explanation why this did not work that way, any hints
are welcome!
Eike
On Thu, Oct 23, 2014 at 8:11 PM, Jason Wee <[email protected]> wrote:
> It could tell more if you have the memory error. We are using version 0.90
> as well and recently our traffic surge and our problem was due to the file
> descriptor reach, based on the error in the log, so we increase the file
> descriptor for the user, max number of processes per user.
>
> This does not mean it would work for your situation but just saying you
> can start to investigate from the log.
>
> Hope it helps.
>
> Jason
>
> On Fri, Oct 24, 2014 at 2:00 AM, Eike Dehling <[email protected]>
> wrote:
>
>> Hi ivan,
>>
>> Thanks for the quick reply.
>>
>> We are doing a lot of aggregations. And those are often the direct cause
>> for a crash: a particularly large aggregation using more than available
>> memory.
>>
>> Field data is around 10gb, of the 31gb assigned to each node, field data
>> does not change much after our initial warmers are loaded.
>>
>> I had expected that the recent addition of two nodes to the cluster would
>> give us enough resources to run trouble-free. Each node now has less shards
>> (from 6 nodes with 4 shards, to 8 nodes with 3 shards), and as such
>> searches and aggregates over one quarter less data? Is this simple math
>> true, or am i missing something?
>>
>> How is field data expected to grow or shrink when going from 4 to 3
>> shards on a node? Does it use about one quarter less memory?
>>
>> Eike
>>
>> On Thursday, October 23, 2014, Ivan Brusic <[email protected]> wrote:
>>
>>> There is more to the issue than merely your configuration. What are your
>>> queries? Are you doing a lot of aggregations, especially on on
>>> high-cardinality fields. What kind of hardware are you running now?
>>>
>>> Using the API, looks at your field cache usage. The field cache is held
>>> within the Java heap space, which would cause the out of memory issues.
>>>
>>> Cheers,
>>>
>>> Ivan
>>>
>>> On Thu, Oct 23, 2014 at 1:26 PM, Eike Dehling <[email protected]>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> We are using elasticsearch in our production backing dashboards with
>>>> social media data. We are running 0.90 version.
>>>>
>>>> Our volume of data grows every day, so we semi-regularly add servers to
>>>> our cluster to keep things running smooth. A year ago we went from 4 nodes
>>>> to 6 nodes. One month ago we have gone from 6 to 8 nodes. Only now, we are
>>>> seeing OutOfMemory like issues again.
>>>>
>>>> My question: It surprises me that we are hitting resource limits again
>>>> so soon, increase in data does not explain that very well. Any suggestions
>>>> for causes?
>>>>
>>>> Best regards,
>>>> Eike dehling
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/elasticsearch/b7ca0070-84bc-4325-8a36-f8600289ae65%40googlegroups.com
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/elasticsearch/w7aXHMa8Sls/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAnMdwVi-xq3oXEsYKROtcDwhR58pnZFsShLhmXvVg5VA%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAnMdwVi-xq3oXEsYKROtcDwhR58pnZFsShLhmXvVg5VA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> --
>> Met vriendelijke groet,
>> Kind regards,
>>
>> Eike Dehling
>> Lead Developer
>>
>> Buzzcapture
>> Herengracht 180, 1016 BR, Amsterdam
>>
>> T: +31 (0)20 3200377
>> M: +31 (0)6 45144840
>>
>> LinkedIn | @buzzcapture
>>
>> Recent
>> 01-08-2014: Buzzcapture breidt uit! Bekijk de vacatures
>> <https://www.buzzcapture.com/over-ons/werken-bij/>op de site
>> 30-07-2014: Tips voor je contentkalender
>> <https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/>
>> van
>> augustus en leestips
>> <https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/>
>> voor
>> op het strand
>> 01-07-2014: EY en Tennet starten met Buzzcapture
>> 28-06-2014: Ziekenhuizen LUMC en MUMC aan de slag met Buzzcapture
>> 01-06-2014: Buzzcapture integreert print
>> <https://www.buzzcapture.com/blog/buzzcapture-biedt-en-online-geintegreerde-monitoring-aan/>
>> content
>> in Social Media Dashboard
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAMp7VWTUk_iVpdFh9yNZWDFv-vz%3DWOW0m3ut-%2B2uXeL3OGgtFQ%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elasticsearch/CAMp7VWTUk_iVpdFh9yNZWDFv-vz%3DWOW0m3ut-%2B2uXeL3OGgtFQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/w7aXHMa8Sls/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAHO4itw28qqSeDs5tDHQcROXX0TZLF6%2BY4k%3DXjQizRjPs4TR%2Bw%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAHO4itw28qqSeDs5tDHQcROXX0TZLF6%2BY4k%3DXjQizRjPs4TR%2Bw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
Met vriendelijke groet,
Kind regards,
Eike Dehling
Lead Developer
Buzzcapture
Herengracht 180, 1016 BR, Amsterdam
T: +31 (0)20 3200377
M: +31 (0)6 45144840
LinkedIn | @buzzcapture
Recent
01-08-2014: Buzzcapture breidt uit! Bekijk de vacatures
<https://www.buzzcapture.com/over-ons/werken-bij/>op de site
30-07-2014: Tips voor je contentkalender
<https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/>
van
augustus en leestips
<https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/>
voor
op het strand
01-07-2014: EY en Tennet starten met Buzzcapture
28-06-2014: Ziekenhuizen LUMC en MUMC aan de slag met Buzzcapture
01-06-2014: Buzzcapture integreert print
<https://www.buzzcapture.com/blog/buzzcapture-biedt-en-online-geintegreerde-monitoring-aan/>
content
in Social Media Dashboard
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAMp7VWR7xtVRutY%3DXkZSbpk41KT8r_NBjKihGU1HGYbd1nK5sQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
