Hello again, setting bootstrap.mlockall to true seems to have made memory usage slower, so like at the place of elasticsearch being killed after ~2 hours it will be killed after ~3 hours. What I see weird, is why is the process releasing memory one back to the OS but not doing it again? And why is it not abiding by this DIRECT_SIZE setting too.
Thanks for the help - - - - - - - - - - Sincerely: Hicham Mallah Software Developer [email protected] 00961 700 49 600 On Thu, Mar 13, 2014 at 4:45 PM, Hicham Mallah <[email protected]>wrote: > Jorg the issue is after the JVM giving back memory to the OS, it starts > going up again, and never gives back memory till its killed, currently > memory usage is up to 66% and still going up. HEAP size is currently set to > 8gb which is 1/4 the amount of memory I have. I tried it at 16, 12, now at > 8 but still facing the issue, lowering it more will cause undesirable speed > on the website. I'll try mlockall now, and see what happens, but looking at > Bigdesk on 18.6mb of swap is used. > > I'll let you know what happens with mlockall on. > > - - - - - - - - - - > Sincerely: > Hicham Mallah > Software Developer > [email protected] > 00961 700 49 600 > > > > On Thu, Mar 13, 2014 at 4:38 PM, [email protected] < > [email protected]> wrote: > >> From the gist, it alls looks very well. There is no reason for the OOM >> killer to kick in. Your system is idle and there is much room for >> everything. >> >> Just to quote you: >> >> "What's happening is that elasticsearch starts using memory till 50% then >> it goes back down to about 30% gradually then starts to go up again >> gradually and never goes back down." >> >> What you see is ES JVM process giving back memory to the OS, which is no >> reason to worry about in regard to process killing. It is just undesirable >> behaviour, and it is all a matter of correct configuration of the heap size. >> >> You should check if your ES starts from service wrapper or from the bin >> folder, and adjust the parameters for heap size. I recommend only to use >> ES_HEAP_SIZE parameter. Set this to max. 50% RAM (as you did). But do not >> use different values at other places, or use MIN or MAX. ES_HEAP_SIZE is >> doing the right thing for you. >> >> With bootstrap mlockall, you can lock the ES JVM process into main >> memory, this helps much regarding to performance and fast GC, as it reduces >> swapping. You can test if this setting will invoke the OOM killer too, as >> it increases the pressure on main memory (but, as said, there is plenty >> room in your machine). >> >> Jörg >> >> >> On Thu, Mar 13, 2014 at 3:13 PM, Hicham Mallah >> <[email protected]>wrote: >> >>> Hello Zachary, >>> >>> Thanks for your reply and the pointer to the settings. >>> >>> Here are the output of the commands you requested: >>> >>> >>> curl -XGET "http://localhost:9200/_nodes/stats"; >>> curl -XGET "http://localhost:9200/_nodes"; >>> >>> https://gist.github.com/codebird/9529114 >>> >>> >>> - - - - - - - - - - >>> Sincerely: >>> Hicham Mallah >>> Software Developer >>> [email protected] >>> 00961 700 49 600 >>> >>> >>> >>> On Thu, Mar 13, 2014 at 3:57 PM, Zachary Tong <[email protected]>wrote: >>> >>>> Can you gist up the output of these two commands? >>>> >>>> curl -XGET "http://localhost:9200/_nodes/stats"; >>>> >>>> curl -XGET "http://localhost:9200/_nodes"; >>>> >>>> Those are my first-stop APIs for determining where memory is being >>>> allocated. >>>> >>>> >>>> By the way, these settings don't do anything anymore (they were >>>> depreciated and removed): >>>> >>>> index.cache.field.type: soft >>>> index.term_index_interval: 256 >>>> index.term_index_divisor: 5 >>>> >>>> index.cache.field.max_size: 10000 >>>> >>>> >>>> >>>> `max_size` was replaced with `indices.fielddata.cache.size` and accepts >>>> a value like "10gb" or "30%" >>>> >>>> And this is just bad settings in general (causes a lot of GC thrashing): >>>> >>>> index.cache.field.expire: 10m >>>> >>>> >>>> >>>> >>>> On Thursday, March 13, 2014 8:42:54 AM UTC-4, Hicham Mallah wrote: >>>> >>>>> Now the process went back down to 25% usage, from now on it will go >>>>> back up, and won't stop going up. >>>>> >>>>> Sorry for spamming >>>>> >>>>> - - - - - - - - - - >>>>> Sincerely: >>>>> Hicham Mallah >>>>> Software Developer >>>>> [email protected] >>>>> 00961 700 49 600 >>>>> >>>>> >>>>> >>>>> On Thu, Mar 13, 2014 at 2:37 PM, Hicham Mallah <[email protected]>wrote: >>>>> >>>>>> Here's the top after ~1 hour running: >>>>>> >>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>>> 780 root 20 0 317g 14g 7.1g S 492.9 46.4 157:50.89 java >>>>>> >>>>>> >>>>>> - - - - - - - - - - >>>>>> Sincerely: >>>>>> Hicham Mallah >>>>>> Software Developer >>>>>> [email protected] >>>>>> 00961 700 49 600 >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Mar 13, 2014 at 2:36 PM, Hicham Mallah >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> Hello Jörg >>>>>>> >>>>>>> Thanks for the reply, our swap size is 2g. I don't know at what % >>>>>>> the process is being killed as the first time it happened I wasn't >>>>>>> around, >>>>>>> and then I never let that happen again as the website is online. After 2 >>>>>>> hours of running the memory in sure is going up to 60%, I am restarting >>>>>>> each time when it arrives at 70% (2h/2h30) when I am around and testing >>>>>>> config changes. When I am not around, I am setting a cron job to restart >>>>>>> the server every 2 hours. Server has apache and mysql running on it too. >>>>>>> >>>>>>> >>>>>>> >>>>>>> - - - - - - - - - - >>>>>>> Sincerely: >>>>>>> Hicham Mallah >>>>>>> Software Developer >>>>>>> [email protected] >>>>>>> 00961 700 49 600 >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 13, 2014 at 2:22 PM, [email protected] < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> You wrote, the OOM killer killed the ES process. With 32g (and the >>>>>>>> swap size), the process must be very big. much more than you >>>>>>>> configured. >>>>>>>> Can you give more info about the live size of the process, after ~2 >>>>>>>> hours? >>>>>>>> Are there more application processes on the box? >>>>>>>> >>>>>>>> Jörg >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Mar 13, 2014 at 12:46 PM, Hicham Mallah < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I have been using elasticsearch on a ubuntu server for a year now, >>>>>>>>> and everything was going great. I had an index of 150,000,000 entries >>>>>>>>> of >>>>>>>>> domain names, running small queries on it, just filtering by 1 term no >>>>>>>>> sorting no wildcard nothing. Now we moved servers, I have now a >>>>>>>>> CentOS 6 >>>>>>>>> server, 32GB ram and running elasticserach but now we have 2 indices, >>>>>>>>> of >>>>>>>>> about 150 million entries each 32 shards, still running the same >>>>>>>>> queries on >>>>>>>>> them nothing changed in the queries. But since we went online with >>>>>>>>> the new >>>>>>>>> server, I have to restart elasticsearch every 2 hours before OOM >>>>>>>>> killer >>>>>>>>> kills it. >>>>>>>>> >>>>>>>>> What's happening is that elasticsearch starts using memory till >>>>>>>>> 50% then it goes back down to about 30% gradually then starts to go up >>>>>>>>> again gradually and never goes back down. >>>>>>>>> >>>>>>>>> I have tried all the solutions I found on the net, I am a >>>>>>>>> developer not a server admin. >>>>>>>>> >>>>>>>>> *I have these setting in my service wrapper configuration* >>>>>>>>> >>>>>>>>> set.default.ES_HOME=/home/elasticsearch >>>>>>>>> set.default.ES_HEAP_SIZE=8192 >>>>>>>>> set.default.MAX_OPEN_FILES=65535 >>>>>>>>> set.default.MAX_LOCKED_MEMORY=10240 >>>>>>>>> set.default.CONF_DIR=/home/elasticsearch/conf >>>>>>>>> set.default.WORK_DIR=/home/elasticsearch/tmp >>>>>>>>> set.default.DIRECT_SIZE=4g >>>>>>>>> >>>>>>>>> # Java Additional Parameters >>>>>>>>> wrapper.java.additional.1=-Delasticsearch-service >>>>>>>>> wrapper.java.additional.2=-Des.path.home=%ES_HOME% >>>>>>>>> wrapper.java.additional.3=-Xss256k >>>>>>>>> wrapper.java.additional.4=-XX:+UseParNewGC >>>>>>>>> wrapper.java.additional.5=-XX:+UseConcMarkSweepGC >>>>>>>>> wrapper.java.additional.6=-XX:CMSInitiatingOccupancyFraction=75 >>>>>>>>> wrapper.java.additional.7=-XX:+UseCMSInitiatingOccupancyOnly >>>>>>>>> wrapper.java.additional.8=-XX:+HeapDumpOnOutOfMemoryError >>>>>>>>> wrapper.java.additional.9=-Djava.awt.headless=true >>>>>>>>> wrapper.java.additional.10=-XX:MinHeapFreeRatio=40 >>>>>>>>> wrapper.java.additional.11=-XX:MaxHeapFreeRatio=70 >>>>>>>>> wrapper.java.additional.12=-XX:CMSInitiatingOccupancyFraction=75 >>>>>>>>> wrapper.java.additional.13=-XX:+UseCMSInitiatingOccupancyOnly >>>>>>>>> wrapper.java.additional.15=-XX:MaxDirectMemorySize=4g >>>>>>>>> # Initial Java Heap Size (in MB) >>>>>>>>> wrapper.java.initmemory=%ES_HEAP_SIZE% >>>>>>>>> >>>>>>>>> *And these in elasticsearch.yml* >>>>>>>>> ES_MIN_MEM: 5g >>>>>>>>> ES_MAX_MEM: 5g >>>>>>>>> #index.store.type=mmapfs >>>>>>>>> index.cache.field.type: soft >>>>>>>>> index.cache.field.max_size: 10000 >>>>>>>>> index.cache.field.expire: 10m >>>>>>>>> index.term_index_interval: 256 >>>>>>>>> index.term_index_divisor: 5 >>>>>>>>> >>>>>>>>> *java version: * >>>>>>>>> java version "1.7.0_51" >>>>>>>>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13) >>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) >>>>>>>>> >>>>>>>>> *Elasticsearch version* >>>>>>>>> "version" : { >>>>>>>>> "number" : "1.0.0", >>>>>>>>> "build_hash" : "a46900e9c72c0a623d71b54016357d5f94c8ea32", >>>>>>>>> "build_timestamp" : "2014-02-12T16:18:34Z", >>>>>>>>> "build_snapshot" : false, >>>>>>>>> "lucene_version" : "4.6" >>>>>>>>> } >>>>>>>>> >>>>>>>>> Using elastica PHP >>>>>>>>> >>>>>>>>> >>>>>>>>> I have tried playing with values up and down to try to make it >>>>>>>>> work, but nothing is changing. >>>>>>>>> >>>>>>>>> Please any help would be highly appreciated. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "elasticsearch" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/4059bf32- >>>>>>>>> ae30-45fa-947c-98ef4540920a%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/4059bf32-ae30-45fa-947c-98ef4540920a%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>>>>> topic/elasticsearch/D4WNQZSvqSU/unsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>> to [email protected]. >>>>>>>> >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/ >>>>>>>> CAKdsXoFcdFx98JugN7oDD0%3DBXMrY5v8-1LtBMdHeAXWJeho67Q% >>>>>>>> 40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcdFx98JugN7oDD0%3DBXMrY5v8-1LtBMdHeAXWJeho67Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/D4WNQZSvqSU/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/f40c285f-36cb-4062-8ee8-db848503c051%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/f40c285f-36cb-4062-8ee8-db848503c051%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAJf9Rn8EZkKCfQ5Pbi-UgXjVWF0OyPnreAFyy%2ByX5Njf70%2B4-g%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAJf9Rn8EZkKCfQ5Pbi-UgXjVWF0OyPnreAFyy%2ByX5Njf70%2B4-g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/D4WNQZSvqSU/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-oJ5Fnjeawyv73FDGrdzcKEWaCT0BtMi84Eb%3DuFUT3w%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-oJ5Fnjeawyv73FDGrdzcKEWaCT0BtMi84Eb%3DuFUT3w%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJf9Rn-HNpespS8roqapVZiz2iZxjWRwXuKYTwgAJsd_8goHiQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
