Jorg the issue is after the JVM giving back memory to the OS, it starts going up again, and never gives back memory till its killed, currently memory usage is up to 66% and still going up. HEAP size is currently set to 8gb which is 1/4 the amount of memory I have. I tried it at 16, 12, now at 8 but still facing the issue, lowering it more will cause undesirable speed on the website. I'll try mlockall now, and see what happens, but looking at Bigdesk on 18.6mb of swap is used.
I'll let you know what happens with mlockall on. - - - - - - - - - - Sincerely: Hicham Mallah Software Developer [email protected] 00961 700 49 600 On Thu, Mar 13, 2014 at 4:38 PM, [email protected] < [email protected]> wrote: > From the gist, it alls looks very well. There is no reason for the OOM > killer to kick in. Your system is idle and there is much room for > everything. > > Just to quote you: > > "What's happening is that elasticsearch starts using memory till 50% then > it goes back down to about 30% gradually then starts to go up again > gradually and never goes back down." > > What you see is ES JVM process giving back memory to the OS, which is no > reason to worry about in regard to process killing. It is just undesirable > behaviour, and it is all a matter of correct configuration of the heap size. > > You should check if your ES starts from service wrapper or from the bin > folder, and adjust the parameters for heap size. I recommend only to use > ES_HEAP_SIZE parameter. Set this to max. 50% RAM (as you did). But do not > use different values at other places, or use MIN or MAX. ES_HEAP_SIZE is > doing the right thing for you. > > With bootstrap mlockall, you can lock the ES JVM process into main memory, > this helps much regarding to performance and fast GC, as it reduces > swapping. You can test if this setting will invoke the OOM killer too, as > it increases the pressure on main memory (but, as said, there is plenty > room in your machine). > > Jörg > > > On Thu, Mar 13, 2014 at 3:13 PM, Hicham Mallah <[email protected]>wrote: > >> Hello Zachary, >> >> Thanks for your reply and the pointer to the settings. >> >> Here are the output of the commands you requested: >> >> >> curl -XGET "http://localhost:9200/_nodes/stats"; >> curl -XGET "http://localhost:9200/_nodes"; >> >> https://gist.github.com/codebird/9529114 >> >> >> - - - - - - - - - - >> Sincerely: >> Hicham Mallah >> Software Developer >> [email protected] >> 00961 700 49 600 >> >> >> >> On Thu, Mar 13, 2014 at 3:57 PM, Zachary Tong <[email protected]>wrote: >> >>> Can you gist up the output of these two commands? >>> >>> curl -XGET "http://localhost:9200/_nodes/stats"; >>> >>> curl -XGET "http://localhost:9200/_nodes"; >>> >>> Those are my first-stop APIs for determining where memory is being >>> allocated. >>> >>> >>> By the way, these settings don't do anything anymore (they were >>> depreciated and removed): >>> >>> index.cache.field.type: soft >>> index.term_index_interval: 256 >>> index.term_index_divisor: 5 >>> >>> index.cache.field.max_size: 10000 >>> >>> >>> >>> `max_size` was replaced with `indices.fielddata.cache.size` and accepts >>> a value like "10gb" or "30%" >>> >>> And this is just bad settings in general (causes a lot of GC thrashing): >>> >>> index.cache.field.expire: 10m >>> >>> >>> >>> >>> On Thursday, March 13, 2014 8:42:54 AM UTC-4, Hicham Mallah wrote: >>> >>>> Now the process went back down to 25% usage, from now on it will go >>>> back up, and won't stop going up. >>>> >>>> Sorry for spamming >>>> >>>> - - - - - - - - - - >>>> Sincerely: >>>> Hicham Mallah >>>> Software Developer >>>> [email protected] >>>> 00961 700 49 600 >>>> >>>> >>>> >>>> On Thu, Mar 13, 2014 at 2:37 PM, Hicham Mallah <[email protected]>wrote: >>>> >>>>> Here's the top after ~1 hour running: >>>>> >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>> 780 root 20 0 317g 14g 7.1g S 492.9 46.4 157:50.89 java >>>>> >>>>> >>>>> - - - - - - - - - - >>>>> Sincerely: >>>>> Hicham Mallah >>>>> Software Developer >>>>> [email protected] >>>>> 00961 700 49 600 >>>>> >>>>> >>>>> >>>>> On Thu, Mar 13, 2014 at 2:36 PM, Hicham Mallah <[email protected]>wrote: >>>>> >>>>>> Hello Jörg >>>>>> >>>>>> Thanks for the reply, our swap size is 2g. I don't know at what % the >>>>>> process is being killed as the first time it happened I wasn't around, >>>>>> and >>>>>> then I never let that happen again as the website is online. After 2 >>>>>> hours >>>>>> of running the memory in sure is going up to 60%, I am restarting each >>>>>> time >>>>>> when it arrives at 70% (2h/2h30) when I am around and testing config >>>>>> changes. When I am not around, I am setting a cron job to restart the >>>>>> server every 2 hours. Server has apache and mysql running on it too. >>>>>> >>>>>> >>>>>> >>>>>> - - - - - - - - - - >>>>>> Sincerely: >>>>>> Hicham Mallah >>>>>> Software Developer >>>>>> [email protected] >>>>>> 00961 700 49 600 >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Mar 13, 2014 at 2:22 PM, [email protected] < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> You wrote, the OOM killer killed the ES process. With 32g (and the >>>>>>> swap size), the process must be very big. much more than you configured. >>>>>>> Can you give more info about the live size of the process, after ~2 >>>>>>> hours? >>>>>>> Are there more application processes on the box? >>>>>>> >>>>>>> Jörg >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 13, 2014 at 12:46 PM, Hicham Mallah <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I have been using elasticsearch on a ubuntu server for a year now, >>>>>>>> and everything was going great. I had an index of 150,000,000 entries >>>>>>>> of >>>>>>>> domain names, running small queries on it, just filtering by 1 term no >>>>>>>> sorting no wildcard nothing. Now we moved servers, I have now a CentOS >>>>>>>> 6 >>>>>>>> server, 32GB ram and running elasticserach but now we have 2 indices, >>>>>>>> of >>>>>>>> about 150 million entries each 32 shards, still running the same >>>>>>>> queries on >>>>>>>> them nothing changed in the queries. But since we went online with the >>>>>>>> new >>>>>>>> server, I have to restart elasticsearch every 2 hours before OOM killer >>>>>>>> kills it. >>>>>>>> >>>>>>>> What's happening is that elasticsearch starts using memory till 50% >>>>>>>> then it goes back down to about 30% gradually then starts to go up >>>>>>>> again >>>>>>>> gradually and never goes back down. >>>>>>>> >>>>>>>> I have tried all the solutions I found on the net, I am a developer >>>>>>>> not a server admin. >>>>>>>> >>>>>>>> *I have these setting in my service wrapper configuration* >>>>>>>> >>>>>>>> set.default.ES_HOME=/home/elasticsearch >>>>>>>> set.default.ES_HEAP_SIZE=8192 >>>>>>>> set.default.MAX_OPEN_FILES=65535 >>>>>>>> set.default.MAX_LOCKED_MEMORY=10240 >>>>>>>> set.default.CONF_DIR=/home/elasticsearch/conf >>>>>>>> set.default.WORK_DIR=/home/elasticsearch/tmp >>>>>>>> set.default.DIRECT_SIZE=4g >>>>>>>> >>>>>>>> # Java Additional Parameters >>>>>>>> wrapper.java.additional.1=-Delasticsearch-service >>>>>>>> wrapper.java.additional.2=-Des.path.home=%ES_HOME% >>>>>>>> wrapper.java.additional.3=-Xss256k >>>>>>>> wrapper.java.additional.4=-XX:+UseParNewGC >>>>>>>> wrapper.java.additional.5=-XX:+UseConcMarkSweepGC >>>>>>>> wrapper.java.additional.6=-XX:CMSInitiatingOccupancyFraction=75 >>>>>>>> wrapper.java.additional.7=-XX:+UseCMSInitiatingOccupancyOnly >>>>>>>> wrapper.java.additional.8=-XX:+HeapDumpOnOutOfMemoryError >>>>>>>> wrapper.java.additional.9=-Djava.awt.headless=true >>>>>>>> wrapper.java.additional.10=-XX:MinHeapFreeRatio=40 >>>>>>>> wrapper.java.additional.11=-XX:MaxHeapFreeRatio=70 >>>>>>>> wrapper.java.additional.12=-XX:CMSInitiatingOccupancyFraction=75 >>>>>>>> wrapper.java.additional.13=-XX:+UseCMSInitiatingOccupancyOnly >>>>>>>> wrapper.java.additional.15=-XX:MaxDirectMemorySize=4g >>>>>>>> # Initial Java Heap Size (in MB) >>>>>>>> wrapper.java.initmemory=%ES_HEAP_SIZE% >>>>>>>> >>>>>>>> *And these in elasticsearch.yml* >>>>>>>> ES_MIN_MEM: 5g >>>>>>>> ES_MAX_MEM: 5g >>>>>>>> #index.store.type=mmapfs >>>>>>>> index.cache.field.type: soft >>>>>>>> index.cache.field.max_size: 10000 >>>>>>>> index.cache.field.expire: 10m >>>>>>>> index.term_index_interval: 256 >>>>>>>> index.term_index_divisor: 5 >>>>>>>> >>>>>>>> *java version: * >>>>>>>> java version "1.7.0_51" >>>>>>>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13) >>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) >>>>>>>> >>>>>>>> *Elasticsearch version* >>>>>>>> "version" : { >>>>>>>> "number" : "1.0.0", >>>>>>>> "build_hash" : "a46900e9c72c0a623d71b54016357d5f94c8ea32", >>>>>>>> "build_timestamp" : "2014-02-12T16:18:34Z", >>>>>>>> "build_snapshot" : false, >>>>>>>> "lucene_version" : "4.6" >>>>>>>> } >>>>>>>> >>>>>>>> Using elastica PHP >>>>>>>> >>>>>>>> >>>>>>>> I have tried playing with values up and down to try to make it >>>>>>>> work, but nothing is changing. >>>>>>>> >>>>>>>> Please any help would be highly appreciated. >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/4059bf32- >>>>>>>> ae30-45fa-947c-98ef4540920a%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/4059bf32-ae30-45fa-947c-98ef4540920a%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "elasticsearch" group. >>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>>>> topic/elasticsearch/D4WNQZSvqSU/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected]. >>>>>>> >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/ >>>>>>> CAKdsXoFcdFx98JugN7oDD0%3DBXMrY5v8-1LtBMdHeAXWJeho67Q% >>>>>>> 40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcdFx98JugN7oDD0%3DBXMrY5v8-1LtBMdHeAXWJeho67Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>> >>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/D4WNQZSvqSU/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/f40c285f-36cb-4062-8ee8-db848503c051%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/f40c285f-36cb-4062-8ee8-db848503c051%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAJf9Rn8EZkKCfQ5Pbi-UgXjVWF0OyPnreAFyy%2ByX5Njf70%2B4-g%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAJf9Rn8EZkKCfQ5Pbi-UgXjVWF0OyPnreAFyy%2ByX5Njf70%2B4-g%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/D4WNQZSvqSU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-oJ5Fnjeawyv73FDGrdzcKEWaCT0BtMi84Eb%3DuFUT3w%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-oJ5Fnjeawyv73FDGrdzcKEWaCT0BtMi84Eb%3DuFUT3w%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJf9Rn9jNBBzyCdjwQQzZvCvkuZ0gyLNdgTCFnbEWcsUYViQug%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
