Just to close this out - we disabled EC2's health checks and spent some time tuning the batch thread-pool size to prevent overrunning the cluster once the memory map cache size exceeds available physical memory. This was successful (we're restricted to a surprisingly small threadpool size of 3).
Thanks. On Saturday, March 14, 2015 at 1:05:49 PM UTC-7, Mark Walkom wrote: > > Can you provide more info on what the error/problem is, logs might help. > > On 14 March 2015 at 10:12, [email protected] <javascript:> < > [email protected] <javascript:>> wrote: > >> I'm out - no experience with EC2. I avoid foreign servers at all cost. >> Maybe 120G RAM is affected by swap/memory overcommit. Do not forget to >> check memlock and memory ballooning. The chances are few you can control >> host settings as a guest in a virtual server environment. >> >> Jörg >> >> On Sat, Mar 14, 2015 at 5:06 PM, Lindsey Poole <[email protected] >> <javascript:>> wrote: >> >>> btw - we're on EC2 I2-4xl hosts, so we have ~120g ram and SSDs. >>> >>> >>> On Saturday, March 14, 2015 at 9:04:34 AM UTC-7, Lindsey Poole wrote: >>>> >>>> I did see the ES_DIRECT_SIZE, but it seems to be ineffective. >>>> >>>> I will try setting -XX:MaxDirectMemorySize directly. >>>> >>>> On Saturday, March 14, 2015 at 4:43:22 AM UTC-7, Jörg Prante wrote: >>>>> >>>>> You may try limit direct memory on JVM level by >>>>> using -XX:MaxDirectMemorySize (default is unlimited). See also >>>>> ES_DIRECT_SIZE in http://www.elastic.co/guide/en/elasticsearch/ >>>>> reference/current/setup-service.html#_linux >>>>> >>>>> I recommend at least 2GB >>>>> >>>>> Jörg >>>>> >>>>> On Sat, Mar 14, 2015 at 1:03 AM, Lindsey Poole <[email protected]> >>>>> wrote: >>>>> >>>>>> Hey guys, >>>>>> >>>>>> We're running into some problems under heavy write, nominal read >>>>>> volume when the Lucene memory mapped files have exhausted available >>>>>> physical memory, and segments from disk must be paged into memory. >>>>>> >>>>>> Are there any configs available to control how much physical memory >>>>>> is available to MemoryMapDirectory? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>>> msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5% >>>>>> 40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/cf50ed46-cacf-414b-8b20-b82595dc2fd0%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/cf50ed46-cacf-414b-8b20-b82595dc2fd0%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHY_M2BAywrG%2BaNcg59xA4_ocph1oqE0bzba4HTqrdLqQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHY_M2BAywrG%2BaNcg59xA4_ocph1oqE0bzba4HTqrdLqQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4bfeb6fa-5fc4-408c-a888-47553ed00738%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
