The advisory of "a lot of heap" means, give as much heap as the JVM is able to process efficiently. There is an upper limit due to JVM engineering state of today. You will not find JVMs that can efficiently manage heaps >32G (except rare expensive commercial JVM products). By efficient I mean GC stalls under a second. There is heavy engineering going on, known as the Shenandoah project, to tackle heaps over 100G by millisecond GC: http://openjdk.java.net/jeps/189
The mere index size is not related to heap size choice. You need large heap if you want filter caching and aggregations/facets cached. Example: I have 350G on index files. On my 3x64G RAM nodes I have assigned 3x16G heap and I do not cache filters, due to the nature of my queries. The other ~48G I left to OS, for file system buffers (direct I/O is the key to fast systems). If I assigned 32G to heap, GC would be inacceptable high, and system would go sluggish after some days, as you had described. It is not a matter of heap size, but of balancing things carefully out between JVM management abilities and operating system I/O power. The challenge is that many ES workload patterns require different balancings. Jörg On Tue, Apr 15, 2014 at 3:42 PM, Wouter van Atteveldt < [email protected]> wrote: > > > > On Tue, Apr 15, 2014 at 2:00 PM, [email protected] < > [email protected]> wrote: > >> This is not Elasticsearch related. If you use a 40g heap of such extreme >> size, you must expect that garbage collection must run for minutes, on >> every JVM I know. >> >> > Right, but it is actually advised to give elastic a lot of heap, right? > The whole index is around 140G, so I would have thought that all frequently > used parts should get loaded in memory, but it still starts running slow > after a while. > > Any ideas? > > -- Wouter > > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CACXi6Xe_Q3xy8xZNo5QDqPpsJot8AknCYXfo5N9Vw2AyOfbVZA%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CACXi6Xe_Q3xy8xZNo5QDqPpsJot8AknCYXfo5N9Vw2AyOfbVZA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoERmpqOqp%2Bzq3jMaCZ7e2OrXuboVP9W6BFU%3DDS51RL%3DPw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
