Yes, kibana will load for whatever you ask for, *but* ES has to maintain index metadata for every index in memory. Those two coupled are pushing things too far for your heap.
Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 12 May 2014 20:45, Abhishek Tiwari <[email protected]> wrote: > How much RAM you need depends on how long you want to keep your data >> around for. So, given you have ~200GB now on 4GB of RAM, you can probably >> extrapolate that out based on your needs. > > > Isn't my problem more with 9G *daily* index, than with total of 200G(20 > days x 9G) indexes? > Correct me if i am wrong here but doesn't kibana ask elasticsearch for > just one day/week of indices(based on the query). > Will elasticsearch really care if i have 500 days of total day-wise > segregated indices out there but am performing queries on *just past 7 > days*? > Is this a total-footprint problem or a daily throughput problem? > > > > On Monday, 12 May 2014 15:30:36 UTC+5:30, Mark Walkom wrote: >> >> It's standard practise to use 50% of system memory for the heap. >> >> How much RAM you need depends on how long you want to keep your data >> around for. So, given you have ~200GB now on 4GB of RAM, you can probably >> extrapolate that out based on your needs. >> >> Regards, >> Mark Walkom >> >> Infrastructure Engineer >> Campaign Monitor >> email: [email protected] >> web: www.campaignmonitor.com >> >> >> On 12 May 2014 19:33, Abhishek Tiwari <[email protected]> wrote: >> >>> add more memory >>> >>> >>> i am doing 15 million docs, which total to ~9G. The average doc size is >>> ~2KB. >>> >>> 1. How much memory would you suggest for my use-case? >>> 2.Also, is it prudent for me to have half of OS memory dedicated to >>> elasticsearch? >>> >>> >>> On Monday, 12 May 2014 14:03:19 UTC+5:30, Mark Walkom wrote: >>>> >>>> You need to reduce your data size, add more memory or add another node. >>>> >>>> Basically, you've reached the limits of that node. >>>> >>>> Regards, >>>> Mark Walkom >>>> >>>> Infrastructure Engineer >>>> Campaign Monitor >>>> email: [email protected] >>>> web: www.campaignmonitor.com >>>> >>>> >>>> On 12 May 2014 16:38, Abhishek Tiwari <[email protected]> wrote: >>>> >>>>> My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem). >>>>> Elasticsearch starts as- >>>>> >>>>> 498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10 >>>>> /usr/bin/java *-Xms4g -Xmx4g -Xss256k* -Djava.awt.headless=true >>>>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: >>>>> CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly >>>>> -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/ >>>>> elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch >>>>> -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/s >>>>> hare/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* >>>>> -Des.default.path.home=/usr/share/elasticsearch >>>>> -Des.default.path.logs=/var/log/elasticsearch >>>>> -Des.default.path.data=/var/lib/elasticsearch >>>>> -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/ >>>>> elasticsearch org.elasticsearch.bootstrap.Elasticsearch >>>>> >>>>> >>>>> The node stopped responding (the ip:9200 status page), and so did >>>>> kibana. It started working fine on a restart. >>>>> i have logstash format docs wherein the index rotates daily. >>>>> Stats: >>>>> Daily: ~11G docs, ~15 million. >>>>> Total: 195G docs, ~300 million. >>>>> >>>>> The logs of the time when it stopped responding are- >>>>> >>>>> [2014-05-12 03:39:08,789][INFO ][cluster.metadata ] [Hannibal >>>>> King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic) >>>>> [2014-05-12 03:40:52,293][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s], >>>>> total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young] >>>>> [150.3mb]->[1.7mb]/[266.2mb]}{[survivor] >>>>> [33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 03:44:11,739][INFO ][cluster.metadata ] [Hannibal >>>>> King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic) >>>>> [2014-05-12 03:45:32,191][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s], >>>>> total [5.2s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young] >>>>> [197.4mb]->[9.3mb]/[266.2mb]}{[survivor] >>>>> [33.2mb]->[0b]/[33.2mb]}{[old] [3.5gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:06:01,224][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], >>>>> total >>>>> [6s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young] >>>>> [134.7mb]->[9.9mb]/[266.2mb]}{[survivor] >>>>> [33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]} >>>>> [2014-05-12 04:08:14,473][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s], >>>>> total [5.8s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young] >>>>> [165.1mb]->[2.7mb]/[266.2mb]}{[survivor] >>>>> [33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:09:07,473][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s], >>>>> total [6.2s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young] >>>>> [265.9mb]->[2.9mb]/[266.2mb]}{[survivor] >>>>> [33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]} >>>>> [2014-05-12 04:10:08,387][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s], >>>>> total [5.4s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young] >>>>> [176.5mb]->[5.8mb]/[266.2mb]}{[survivor] >>>>> [33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]} >>>>> [2014-05-12 04:13:12,774][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s], >>>>> total [5.6s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young] >>>>> [167.4mb]->[12.9mb]/[266.2mb]}{[survivor] >>>>> [33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]} >>>>> [2014-05-12 04:14:22,729][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s], >>>>> total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young] >>>>> [224.2mb]->[3.5mb]/[266.2mb]}{[survivor] >>>>> [33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:15:12,192][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s], >>>>> total [5.2s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young] >>>>> [234mb]->[2.4mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:15:32,344][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s], >>>>> total [6.3s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [1.2mb]->[34.7mb]/[266.2mb]}{[survivor] >>>>> [33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:15:39,627][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s], >>>>> total [6.7s]/[4.7h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [34.7mb]->[45.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:15:51,547][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], >>>>> total >>>>> [6.5s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [24.4mb]->[32.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:16:03,300][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s], >>>>> total [5.8s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young] >>>>> [266.2mb]->[21.9mb]/[266.2mb]}{[survivor] >>>>> [23.4mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:16:12,454][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], >>>>> total >>>>> [6.6s]/[4.7h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [196.8mb]->[33.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:16:24,635][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s], >>>>> total [5.7s]/[4.8h], memory [3.9gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [246.4mb]->[44.3mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:16:32,239][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s], >>>>> total [6.4s]/[4.8h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [221.5mb]->[31.9mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:16:43,990][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s], >>>>> total [6.4s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [40.5mb]->[64.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:16:51,259][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s], >>>>> total [6.6s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [64.2mb]->[74.6mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:16:58,791][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], >>>>> total >>>>> [7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [74.6mb]->[89.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> [2014-05-12 04:17:06,028][INFO ][monitor.jvm ] [Hannibal >>>>> King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s], >>>>> total [6.7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young] >>>>> [89.7mb]->[93.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] >>>>> [3.6gb]->[3.6gb]/[3.6gb]} >>>>> : >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40goo >>>>> glegroups.com<https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70% >>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bT-DR0oYaAcHOxgR0MM7SdoEM%3DXw_GcRD9aGZT4uT4gA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
