Hi Jörg, Running just 1 JVM with 32GB on a 24-core 256GB machine is a waste. CPU, I/O, memory metrics substantiate this. And off course we need to explore multi-instance before asking mgmt for more money.
Regarding memlock: if no contiguous RAM is available I'd expect a fast error and not a totally hanging process and call traces (mentioning a 120 second timeouts) in the dmesg. Do you think this is maybe a jvm or elasticsearch bug? If so, i'll file it. Regards, Renzo Op dinsdag 26 augustus 2014 17:10:56 UTC+2 schreef Jörg Prante: > > You should run one node per host. > > Two nodes add overhead and suffer from the effects you described. > > For mlockall, the user needs privilege to allocate the specified locked > mem, and the OS need contiguous RAM per mlockall call. If the user's > memlock limit is exhausted, or if RAM allocation gets fragmented, > memlocking is no longer possible and fails. > > Jörg > > > On Tue, Aug 26, 2014 at 2:54 PM, R. Toma <[email protected] <javascript:> > > wrote: > >> Hi all, >> >> In an attempt to squeeze more power out of our physical servers we want >> to run multiple ES jvm's per server. >> >> Some specs: >> - servers has 24 cores, 256GB ram >> - each instance binds on different (alias) ip >> - each instance has 32GB heap >> - both instances run under user 'elastic' >> - limits for 'elastic' user: memlock=unlimited >> - es config for both instances: bootstrap.mlockall=true >> >> The 1st instance has been running for weeks. >> >> When starting the 2nd instance the following things happen: >> - increase of overal cpu load >> - lots of I/O to disks >> - no logging for 2nd instance >> - 2nd instance hangs >> - 1st instance keeps running, but gets slowish >> - cd /proc/<pid> causes a hang of cd process (until 2nd instance is >> killed) >> - exec 'ps axuw' causes a hang of ps process (until 2nd instance is >> killed) >> >> Maybe (un)related: I have never been able to run Elasticsearch in a >> virtualbox with memlock=unlimited and mlockall=true. >> >> >> After an hour of trial & errors I found that removing setting >> 'bootstrap.mlockall' (setting it to false) from 2nd instance's >> configuration fixes things. >> >> I am confused, but acknowledge I do not know anything about memlocking. >> >> Any ideas? >> >> Regards, >> Renzo >> >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dbe8fb95-2054-45ac-a07a-5bf2955e8869%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
