[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878056#comment-16878056 ] ASF subversion and git services commented on SOLR-13003: Commit bf1d14539cb0c524e828113cbabd5d896a9eae60 in lucene-solr's branch refs/heads/branch_8x from Andrzej Bialecki [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=bf1d145 ] SOLR-13003: Fix WrapperModel method delegation. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug >Affects Versions: 7.3.1 >Reporter: Cetra Free >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 8.2 > > Attachments: CLRU-logging.patch, SOLR-13003.patch, > lrucacheexpanded.png, lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, > solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878055#comment-16878055 ] ASF subversion and git services commented on SOLR-13003: Commit 66812b2a6758240f401a5c8aee1488a1f4d14d5f in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=66812b2 ] SOLR-13003: Fix WrapperModel method delegation. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug >Affects Versions: 7.3.1 >Reporter: Cetra Free >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 8.2 > > Attachments: CLRU-logging.patch, SOLR-13003.patch, > lrucacheexpanded.png, lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, > solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877960#comment-16877960 ] ASF subversion and git services commented on SOLR-13003: Commit 9d52e5be3b81a8ad3ab22ffc002e10b67dbee28e in lucene-solr's branch refs/heads/branch_8x from Andrzej Bialecki [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=9d52e5b ] SOLR-13003: Query Result Cache does not honour maxRamBytes parameter. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug >Affects Versions: 7.3.1 >Reporter: Cetra Free >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 8.2 > > Attachments: CLRU-logging.patch, SOLR-13003.patch, > lrucacheexpanded.png, lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, > solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877909#comment-16877909 ] ASF subversion and git services commented on SOLR-13003: Commit 5897787291b8bb5c9ffe83280e67418613dcdbfd in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5897787 ] SOLR-13003: Query Result Cache does not honour maxRamBytes parameter. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug >Affects Versions: 7.3.1 >Reporter: Cetra Free >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 8.2 > > Attachments: CLRU-logging.patch, SOLR-13003.patch, > lrucacheexpanded.png, lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, > solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873656#comment-16873656 ] Andrzej Bialecki commented on SOLR-13003: -- This patch includes the following changes: * adds Accountable to FastLRUCache * adds tighter tracking of both key and value sizes in all Solr caches. This builds on the changes in LUCENE-8855. * some refactoring to put common constants in one place and to use more realistic defaults for unknown objects / query types. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug >Affects Versions: 7.3.1 >Reporter: Cetra Free >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 8.1.2 > > Attachments: CLRU-logging.patch, SOLR-13003.patch, > lrucacheexpanded.png, lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, > solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835301#comment-16835301 ] David Smiley commented on SOLR-13003: - I think the lesson here is: for caches with components where the key or value is not Accountable, we should generate an error if you try to size the cache based on the memory size since we can't compute it.. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: CLRU-logging.patch, lrucacheexpanded.png, > lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831577#comment-16831577 ] Erick Erickson commented on SOLR-13003: --- >From Brian Ecker on the user's list, who should get credit in CHANGES whenever >this is fixed for his sleuthing. Just to update here in order to help others that might run into similar issues in the future, the problem is resolved. The issue was caused by the queryResultCache. This was very easy to determine by analyzing a heap dump. In our setup we had the following config: In reality this maxRamMB="3072" was not as expected, and this cache was using *way* more memory (about 6-8 times the amount). See the following screenshot from Eclipse MAT (http://oi63.tinypic.com/epn341.jpg). Notice in the left window that ramBytes, the internal calculation of how much memory Solr currently thinks this cache is using, is 1894333464B (1894MB). Now notice that the highlighted line, the ConcurrentLRUCache used internally by the FastLRUCache representing the queryResultCache, is actually using 12212779160B (12212MB). On further investigation, I realized that this cache is a map from a query with all its associated objects as the key, to a very simple object containing an array of document (integer) ids as the value. Looking into the lucene-solr source, I found the following line for the calculation of ramBytesUsed https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/util/ConcurrentLRUCache.java#L605. Surprisingly, the query objects used as keys in the queryResultCache do not implement Accountable as far as I can tell, and this lines up very well with our observation of memory usage because in the heap dump we can also see that the keys in the cache are using substantially more memory than the values and completely account for the additional memory usage. It was quite surprising to me that the keys were given a default value of 192B as specified in LRUCache.DEFAULT_RAM_BYTES_USED because I can't actually imagine a case where the keys in the queryResultCache would be so small. I imagine that in almost all cases the keys would actually be larger than the values for the queryResultCache, but that's probably not true for all usages of a FastLRUCache. We solved our memory usage issue by drastically reducing the maxRamMB value and calculating the actual max usage as maxRamMB * 8. It would be quite useful to have this detail at least documented somewhere. -Brian > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: CLRU-logging.patch, lrucacheexpanded.png, > lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[
https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695315#comment-16695315
]
Cetra Free commented on SOLR-13003:
---
With regards to {{FastLRUCache}}: the actual accounting implementation is in
{{ConcurrentLRUCache}} which appears to be doing the right thing. But the
issue is the size comparison, so I don't think swapping would help. I will give
it a shot regardless.
I've noticed the QueryResultKey is greater than 75kb of memory per key, which
is much greater size than the value.
I will try out the patched version from Shawn & see if I can trigger it and get
some stats out.
> Query Result Cache does not honour maxRamBytes parameter
>
>
> Key: SOLR-13003
> URL: https://issues.apache.org/jira/browse/SOLR-13003
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 7.3.1
>Reporter: Cetra Free
>Priority: Major
> Attachments: CLRU-logging.patch, lrucacheexpanded.png,
> lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml
>
>
> When using the maxRamBytes parameter with the queryResultCache directive, we
> have seen the retained size of the cache orders of magnitude larger than what
> is configured.
> Please see attached VisualVM output which shows the retained size is about
> 1.5gb, but the maxRamBytes is set to 64mb.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694354#comment-16694354 ] Shalin Shekhar Mangar commented on SOLR-13003: -- The default queryResultCache implementation is the LRUCache which supports maxRamMB and that is tested. You are using the FastLRUCache for query result cache. This combination is not tested. I think that's where the problem lies. Do you mind switching query result cache back to LRUCache. Was there are a reason you made the switch? I'll look into how query result cache using FastLRUCache triggers this bug and either fix it or disable the specific combination. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: CLRU-logging.patch, lrucacheexpanded.png, > lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694230#comment-16694230 ] Shawn Heisey commented on SOLR-13003: - Ouch. The jar is five megabytes! To use it, you'll need to stop Solr, delete the existing solr-core-7.3.1.jar file from server/solr-webapp/webapp/WEB-INF/lib and copy this new jar in its place, then start Solr back up. Before starting Solr again, you will also need to edit your log4j.properties file to increase the rollover size for the logfile from 4MB to something much larger. I expect there to be a LOT of data logged. Then you'll have to do the steps that result in the cache getting much larger than the defined size. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: CLRU-logging.patch, lrucacheexpanded.png, > lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694231#comment-16694231 ] Shawn Heisey commented on SOLR-13003: - I just realized that I have made a mistake in the patch. I will update the patch and the jar. Wait until I get that done before trying it. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: CLRU-logging.patch, lrucacheexpanded.png, > lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694234#comment-16694234 ] Shawn Heisey commented on SOLR-13003: - OK, I was wrong. I thought I had made an error that would result in things being logged for caches that were not defined with the max ram size, but after checking the code more carefully, I have concluded that this will not be a problem. So the attached files are good. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: CLRU-logging.patch, lrucacheexpanded.png, > lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694228#comment-16694228 ] Shawn Heisey commented on SOLR-13003: - Attached a patch for adding logging. Will attach a custom jar when Solr finishes compiling. [~cetra3], you're in the best position to test this. I no longer have access to Solr servers that I can change at will, so it's difficult for me to do this. I would suggest that when testing this, you only have one of Solr's caches set up with maxRamMB, define all the others with the usual "size" parameters. If you could provide a file with a typical search response from Solr so we can see how much data a response contains, that would be helpful. If you choose to do testing with the filterCache, we will need to know how many docs (maxDoc, not numDoc) the core contains. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: CLRU-logging.patch, lrucacheexpanded.png, > lrucachemaxmb.png, solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694226#comment-16694226 ] Erick Erickson commented on SOLR-13003: --- Hmmm, does it stay that size if you force a full GC? I'm wondering if somehow the memory reported is waiting to be collected. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: lrucacheexpanded.png, lrucachemaxmb.png, solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694192#comment-16694192 ] Shawn Heisey commented on SOLR-13003: - ConcurrentLRUCache code is identical between 7.3.1 and master. So there are no bugs fixed after the version you're running. If the entry object in the cache is not an instance of the Accountable interface, then ConcurrentLRUCache assumes it to be 192 bytes. This might be very small compared to actual cache entries. At 192 bytes, 64MB can handle over 300,000 entries. A DocList does implement Accountable. SolrCache and QueryResultKey do not. I'm betting that the DocList instances are what the cache will be using to calculate its size. In that expanded tree that you shared, the "ramBytes" field seems to be saying that the cache is tracking its size as about 7 megabytes. Which suggests that maybe the code to calculate the cache size is not working right. Should probably come up with a custom solr-core jar where ConcurrentLRUCache has extra logging, and see if we can track down where the problem is. It'll get extremely verbose. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: lrucacheexpanded.png, lrucachemaxmb.png, solrconfig.xml > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[
https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694134#comment-16694134
]
Cetra Free commented on SOLR-13003:
---
I've added in this expanded cache line as as screenshot.
My suspicion is that the RAM amount used is not calculated correctly which is
what is causing this issue.
I couldn't find any way to validate that the returned
{{Accountable.ramBytesUsed()}} size is accurate for the cache type
({{SolrCache}}).
The config for caches is as follows (I've attached the full config to this
ticket):
{code:xml}
{code}
> Query Result Cache does not honour maxRamBytes parameter
>
>
> Key: SOLR-13003
> URL: https://issues.apache.org/jira/browse/SOLR-13003
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 7.3.1
>Reporter: Cetra Free
>Priority: Major
> Attachments: lrucacheexpanded.png, lrucachemaxmb.png, solrconfig.xml
>
>
> When using the maxRamBytes parameter with the queryResultCache directive, we
> have seen the retained size of the cache orders of magnitude larger than what
> is configured.
> Please see attached VisualVM output which shows the retained size is about
> 1.5gb, but the maxRamBytes is set to 64mb.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[
https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694114#comment-16694114
]
Shawn Heisey commented on SOLR-13003:
-
ConcurrentLRUCache looks (on first blush) to be coded right, but further
checking might reveal a problem that wasn't immediately apparent.
The rest of this comment is a side note. It's not really connected to the
problem in this issue, but I came across it while looking into the code, so I
wanted to note it.
This bit of code in FastLRUCache looks problematic to me:
{code:java}
if (maxRamBytes != Long.MAX_VALUE) {
int ramLowerWatermark = (int) (maxRamBytes * 0.8);
description = generateDescription(maxRamBytes, ramLowerWatermark,
newThread);
cache = new ConcurrentLRUCache(ramLowerWatermark, maxRamBytes,
newThread, null);
} else {
description = generateDescription(limit, initialSize, minLimit,
acceptableLimit, newThread);
cache = new ConcurrentLRUCache<>(limit, minLimit, acceptableLimit,
initialSize, newThread, false, null);
}
{code}
The reason I think it's problematic: The 80 percent calculation is converted
to an int ... which means that the maximum that the low watermark can be is 2GB
... that's incorrect if you have requested a large enough max size.
I did some test code with that calculation. With an input of 8589934592, which
is 8GB, the value for the 80 percent calculation converted to an int is
2147483647. Far less than 80 percent. I think that ramLowerWatermark should
be a long, and the cast to int shouldn't be there.
> Query Result Cache does not honour maxRamBytes parameter
>
>
> Key: SOLR-13003
> URL: https://issues.apache.org/jira/browse/SOLR-13003
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 7.3.1
>Reporter: Cetra Free
>Priority: Major
> Attachments: lrucachemaxmb.png
>
>
> When using the maxRamBytes parameter with the queryResultCache directive, we
> have seen the retained size of the cache orders of magnitude larger than what
> is configured.
> Please see attached VisualVM output which shows the retained size is about
> 1.5gb, but the maxRamBytes is set to 64mb.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694102#comment-16694102 ] Shawn Heisey commented on SOLR-13003: - What *EXACTLY* do you have in solrconfig.xml? Can you share the entire file? So far, this is sounding like a support issue -- and this issue tracker is not for support requests. I would like to see what is shown if you open up the "cache" line (type ConcurrentLRUCache) -- the one that shows 1.5GB. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: lrucachemaxmb.png > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[ https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693942#comment-16693942 ] Erick Erickson commented on SOLR-13003: --- I'm not sure what that image is showing me. What is maxRamBytes? Just the value of the long in the LRUCache? If so it's exactly correct, you set the parameter in MB and the size reported is in bytes and 64*1024*1024 is exactly the value shown. If you agree, you should close this JIRA, and I'd also recommend in future you post your question on the user's list before raising a JIRA to be sure it's a real code issue rather than a misunderstanding. > Query Result Cache does not honour maxRamBytes parameter > > > Key: SOLR-13003 > URL: https://issues.apache.org/jira/browse/SOLR-13003 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.3.1 >Reporter: Cetra Free >Priority: Major > Attachments: lrucachemaxmb.png > > > When using the maxRamBytes parameter with the queryResultCache directive, we > have seen the retained size of the cache orders of magnitude larger than what > is configured. > Please see attached VisualVM output which shows the retained size is about > 1.5gb, but the maxRamBytes is set to 64mb. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter
[
https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693955#comment-16693955
]
Cetra Free commented on SOLR-13003:
---
Sorry if it was not clear.
The maxRamBytes is set to {{67108864}} bytes (64mb).
The Retained size of the ConcurrentLRUCache is {{1589995450}} bytes (1.5gb)
In other words, the RAM size is about 21x the size of the configured limit.
> Query Result Cache does not honour maxRamBytes parameter
>
>
> Key: SOLR-13003
> URL: https://issues.apache.org/jira/browse/SOLR-13003
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 7.3.1
>Reporter: Cetra Free
>Priority: Major
> Attachments: lrucachemaxmb.png
>
>
> When using the maxRamBytes parameter with the queryResultCache directive, we
> have seen the retained size of the cache orders of magnitude larger than what
> is configured.
> Please see attached VisualVM output which shows the retained size is about
> 1.5gb, but the maxRamBytes is set to 64mb.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
