After I completed the analysis of the fielddata cache in my situation, I sent a longer analysis within Microsoft. A colleague pointed me to the page http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html . There is lots of great info there and I'm really glad that the default size of the fielddata cache is clearly called out as *unbounded*.
There are two serious flaws with that page that I request get fixed. First, regarding this paragraph: *The indices.fielddata.cache.size controls how much heap space is allocated to fielddata. When you run a query that requires access to new field values, it will load the values into memory and then try to add them to fielddata. If the resulting fielddata size would exceed the specified size, then other values would be evicted in order to make space*. The last sentence is true ONLY when all of the 16 segments in the cache are filled up. In reality the total size is divided by 16 and then checked on a segment by segment basis. I have seen posts complaining that they set the fielddata cache size and see evictions even though the cache as a whole isn't filled up to the size they configured. The evictions occur on a segment basis. If you see evictions even though the cache isn't 'full', your use of segments is not evenly spread across all 16 -- not that there is anything you can do about that! I suspect that as ES deployments grow in scale the 16 segments would become more evenly used. Second, there are the lines: *There is another setting which you may see online: indices.fielddata.cache.expire* *We beg that you never use this setting! It will likely be deprecated in the future.* *This setting tells Elasticsearch to evict values from fielddata if they are older than expire, whether the values are being used or not.* I can imagine that the setting might not be a good idea or might even be deprecated, but the reason given is wrong. The expire value and the expiration process in the cache definitely compare to last accessed time -- which is updated every time the value is accessed. Thus if the values are being accessed within the expiry window, they will NOT be evicted. Craig. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/34fceb06-8df8-4ba4-83bf-2969f4561732%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
