Maybe aggregations are a cause for memory problems. According to the docs, we set the fielddata cache property to indices.fielddata.cache.size: 40% ... hoping this will help to avoid this kind of issues.
Regards, Abid Am Mittwoch, 25. März 2015 00:47:09 UTC+1 schrieb Jason Wee: > > Few filters should be fine but aggregations... hm.... > > You could use dump stack trace and/or heap dump if it happen again and > start to analyze from there. Try as well, > > http://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html > > > hth > > jason > > On Tue, Mar 24, 2015 at 11:59 PM, Abid Hussain > <[email protected] <javascript:>> wrote: > > All settings except from ES_HEAP (set to 10 GB) are defaults, so I > actually > > am not sure about the new gen setting. > > > > The host has 80 GB memory in total and 24 CPU cores. All ES indices > together > > sum up to ~32 GB where the biggest indices are of size ~8 GB. > > > > We are using queries mostly together with filters and also aggregations. > > > > We "solved" the problem with a restart of the cluster. Are there any > > recommended diagnostics to be performed when this problem occurs next > time? > > > > Regards, > > > > Abid > > > > Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee: > >> > >> what is the new gen setting? how much is the system memory in total? > >> how many cpu cores in the node? what query do you use? > >> > >> jason > >> > >> On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain > >> <[email protected]> wrote: > >> > Hi all, > >> > > >> > we today got (for the first time) warning messages which seem to > >> > indicate a > >> > memory problem: > >> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm ] [Danger] > >> > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], > total > >> > [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] > >> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] > [149.7mb]->[0b]/[149.7mb]}{[old] > >> > [6.9gb]->[3.7gb]/[8.5gb]} > >> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm ] [Danger] > >> > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], > total > >> > [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] > >> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] > [149.7mb]->[0b]/[149.7mb]}{[old] > >> > [6.9gb]->[3.7gb]/[8.5gb]} > >> > [2015-03-24 09:08:15,372][WARN ][monitor.jvm ] [Danger] > >> > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], > >> > total > >> > [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young] > >> > [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old] > >> > [3.7gb]->[4.9gb]/[8.5gb]} > >> > [2015-03-24 09:08:18,192][WARN ][monitor.jvm ] [Danger] > >> > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], > >> > total > >> > [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young] > >> > [845.4mb]->[1.2mb]/[1.1gb]}{[survivor] > >> > [149.7mb]->[149.7mb]/[149.7mb]}{[old] > >> > [4.9gb]->[6gb]/[8.5gb]} > >> > [2015-03-24 09:08:21,506][WARN ][monitor.jvm ] [Danger] > >> > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], > >> > total > >> > [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young] > >> > [848.6mb]->[2.1mb]/[1.1gb]}{[survivor] > >> > [149.7mb]->[149.7mb]/[149.7mb]}{[old] > >> > [6gb]->[7.2gb]/[8.5gb]} > >> > > >> > We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g, > >> > other > >> > settings are defaults. From previous posts related to this issue, it > is > >> > said > >> > that field data cache may be a problem. > >> > > >> > Requesting /_nodes/stats/indices/fielddata says: > >> > { > >> > "cluster_name": "my_cluster", > >> > "nodes": { > >> > "ILUggMfTSvix8Kc0nfNVAw": { > >> > "timestamp": 1427188716203, > >> > "name": "Danger", > >> > "transport_address": "inet[/192.168.110.91:9300]", > >> > "host": "xxx", > >> > "ip": [ > >> > "inet[/192.168.110.91:9300]", > >> > "NONE" > >> > ], > >> > "indices": { > >> > "fielddata": { > >> > "memory_size_in_bytes": 64822224, > >> > "evictions": 0 > >> > } > >> > } > >> > } > >> > } > >> > } > >> > > >> > Running top results in: > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >> > 12735 root 20 0 15.8g 10g 0 S 74 13.2 2485:26 java > >> > > >> > Any ideas what to do? If possible I would rather avoid increasing > >> > ES_HEAP as > >> > there isn't that much free memory left on the host. > >> > > >> > Regards, > >> > > >> > Abid > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "elasticsearch" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> > an > >> > email to [email protected]. > >> > To view this discussion on the web visit > >> > > >> > > https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com. > > > >> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com. > > > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fbf022ac-1b94-413c-b7db-0bee0ea6dbe8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
