Few filters should be fine but aggregations... hm.... You could use dump stack trace and/or heap dump if it happen again and start to analyze from there. Try as well, http://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html
hth jason On Tue, Mar 24, 2015 at 11:59 PM, Abid Hussain <[email protected]> wrote: > All settings except from ES_HEAP (set to 10 GB) are defaults, so I actually > am not sure about the new gen setting. > > The host has 80 GB memory in total and 24 CPU cores. All ES indices together > sum up to ~32 GB where the biggest indices are of size ~8 GB. > > We are using queries mostly together with filters and also aggregations. > > We "solved" the problem with a restart of the cluster. Are there any > recommended diagnostics to be performed when this problem occurs next time? > > Regards, > > Abid > > Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee: >> >> what is the new gen setting? how much is the system memory in total? >> how many cpu cores in the node? what query do you use? >> >> jason >> >> On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain >> <[email protected]> wrote: >> > Hi all, >> > >> > we today got (for the first time) warning messages which seem to >> > indicate a >> > memory problem: >> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm ] [Danger] >> > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total >> > [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] >> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] >> > [6.9gb]->[3.7gb]/[8.5gb]} >> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm ] [Danger] >> > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total >> > [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] >> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] >> > [6.9gb]->[3.7gb]/[8.5gb]} >> > [2015-03-24 09:08:15,372][WARN ][monitor.jvm ] [Danger] >> > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], >> > total >> > [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young] >> > [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old] >> > [3.7gb]->[4.9gb]/[8.5gb]} >> > [2015-03-24 09:08:18,192][WARN ][monitor.jvm ] [Danger] >> > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], >> > total >> > [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young] >> > [845.4mb]->[1.2mb]/[1.1gb]}{[survivor] >> > [149.7mb]->[149.7mb]/[149.7mb]}{[old] >> > [4.9gb]->[6gb]/[8.5gb]} >> > [2015-03-24 09:08:21,506][WARN ][monitor.jvm ] [Danger] >> > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], >> > total >> > [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young] >> > [848.6mb]->[2.1mb]/[1.1gb]}{[survivor] >> > [149.7mb]->[149.7mb]/[149.7mb]}{[old] >> > [6gb]->[7.2gb]/[8.5gb]} >> > >> > We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g, >> > other >> > settings are defaults. From previous posts related to this issue, it is >> > said >> > that field data cache may be a problem. >> > >> > Requesting /_nodes/stats/indices/fielddata says: >> > { >> > "cluster_name": "my_cluster", >> > "nodes": { >> > "ILUggMfTSvix8Kc0nfNVAw": { >> > "timestamp": 1427188716203, >> > "name": "Danger", >> > "transport_address": "inet[/192.168.110.91:9300]", >> > "host": "xxx", >> > "ip": [ >> > "inet[/192.168.110.91:9300]", >> > "NONE" >> > ], >> > "indices": { >> > "fielddata": { >> > "memory_size_in_bytes": 64822224, >> > "evictions": 0 >> > } >> > } >> > } >> > } >> > } >> > >> > Running top results in: >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > 12735 root 20 0 15.8g 10g 0 S 74 13.2 2485:26 java >> > >> > Any ideas what to do? If possible I would rather avoid increasing >> > ES_HEAP as >> > there isn't that much free memory left on the host. >> > >> > Regards, >> > >> > Abid >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "elasticsearch" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to [email protected]. >> > To view this discussion on the web visit >> > >> > https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itw3Ezh7fxMKaHAJLcK1kPSPhxMOVNNPZX-w0rpPnVmrqg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
