On Thu, Mar 26, 2015 at 2:28 PM, Paweł Róg <pro...@gmail.com> wrote: > > When we sum all of sizes we can see 13.5GB of primitive arrays pointed by > less than 20M references. As we can see ES&Lucene use a lot of arrays of > primitives. >
I think it depends what takes memory in your heap. For instance, fielddata mainly uses very large primitive arrays, so the impact of compressed oops is almost null. The same would be true for aggregations and p/c queries. On the other hand the filter cache happily uses objects in order to model the keys (filter/segment pairs) and values (sets of doc ids). So if you have a large filter cache that contains lots of small entries, compressed oops could help a bit. This might be true for users who have large collections of percolator queries too. Of course we have to remember that the bigger heap means more work for GC > (and currently used in JVM: CMS or G1 are not very efficient for large > heaps), but ... Is there really a magical line (32GB) after crossing we get > into "JVM troubles" or we can find a lot of cases where crossing the > magical boundary makes sense? > Thanks for bringing this up. I agree this is very important and this is a reason why eg. elasticsearch is moving to doc values by default on not_analyzed fields in 2.0: https://github.com/elastic/elasticsearch/pull/10209 (which effectively moves fielddata memory usage from the heap to the OS cache). The main issue about large heaps is when they make major garbage collections long enough so that they look like the node left from a cluster perspective. I don't think there is a magical line since it also depends on the hardware and the complexity of the graph of objects but it is probably something to keep an eye on. -- Adrien -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAh%3D5QLkE5F0edwqZpNPzcQHVuQ1YOku18cBxHvLcWJm8g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
