[
https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831577#comment-16831577
]
Erick Erickson commented on SOLR-13003:
---------------------------------------
>From Brian Ecker on the user's list, who should get credit in CHANGES whenever
>this is fixed for his sleuthing.
Just to update here in order to help others that might run into similar
issues in the future, the problem is resolved. The issue was caused by the
queryResultCache. This was very easy to determine by analyzing a heap dump.
In our setup we had the following config:
<queryResultCache class="solr.FastLRUCache" maxRamMB="3072"
autowarmCount="0"/>
In reality this maxRamMB="3072" was not as expected, and this cache was
using *way* more memory (about 6-8 times the amount). See the following
screenshot from Eclipse MAT (http://oi63.tinypic.com/epn341.jpg). Notice in
the left window that ramBytes, the internal calculation of how much memory
Solr currently thinks this cache is using, is 1894333464B (1894MB). Now
notice that the highlighted line, the ConcurrentLRUCache used internally by
the FastLRUCache representing the queryResultCache, is actually using
12212779160B (12212MB). On further investigation, I realized that this
cache is a map from a query with all its associated objects as the key, to
a very simple object containing an array of document (integer) ids as the
value.
Looking into the lucene-solr source, I found the following line for the
calculation of ramBytesUsed
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/util/ConcurrentLRUCache.java#L605.
Surprisingly, the query objects used as keys in the queryResultCache do not
implement Accountable as far as I can tell, and this lines up very well
with our observation of memory usage because in the heap dump we can also
see that the keys in the cache are using substantially more memory than the
values and completely account for the additional memory usage. It was quite
surprising to me that the keys were given a default value of 192B as
specified in LRUCache.DEFAULT_RAM_BYTES_USED because I can't actually
imagine a case where the keys in the queryResultCache would be so small. I
imagine that in almost all cases the keys would actually be larger than the
values for the queryResultCache, but that's probably not true for all
usages of a FastLRUCache.
We solved our memory usage issue by drastically reducing the maxRamMB value
and calculating the actual max usage as maxRamMB * 8. It would be quite
useful to have this detail at least documented somewhere.
-Brian
> Query Result Cache does not honour maxRamBytes parameter
> --------------------------------------------------------
>
> Key: SOLR-13003
> URL: https://issues.apache.org/jira/browse/SOLR-13003
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Affects Versions: 7.3.1
> Reporter: Cetra Free
> Priority: Major
> Attachments: CLRU-logging.patch, lrucacheexpanded.png,
> lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml
>
>
> When using the maxRamBytes parameter with the queryResultCache directive, we
> have seen the retained size of the cache orders of magnitude larger than what
> is configured.
> Please see attached VisualVM output which shows the retained size is about
> 1.5gb, but the maxRamBytes is set to 64mb.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
