[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[ https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666163#comment-15666163 ] ASF subversion and git services commented on SOLR-9633: --- Commit b57a5e41f82418621d8ac7e42edf0cd08dcee6be in lucene-solr's branch refs/heads/master from [~shalinmangar] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b57a5e4 ] SOLR-9633: Fix issue number in CHANGES.txt > Limit FastLRUCache by RAM Usage > --- > > Key: SOLR-9633 > URL: https://issues.apache.org/jira/browse/SOLR-9633 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shalin Shekhar Mangar > Fix For: master (7.0), 6.4 > > Attachments: SOLR-9633.patch, SOLR-9633.patch > > > SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on > memory usage. That helps with the query result cache but not with the filter > cache which defaults to FastLRUCache. This issue intends to add the same > feature to FastLRUCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[ https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666166#comment-15666166 ] ASF subversion and git services commented on SOLR-9633: --- Commit d3420c31a92180653b767850e82f55ec3d20fda2 in lucene-solr's branch refs/heads/branch_6x from [~shalinmangar] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d3420c3 ] SOLR-9633: Fix issue number in CHANGES.txt (cherry picked from commit b57a5e4) > Limit FastLRUCache by RAM Usage > --- > > Key: SOLR-9633 > URL: https://issues.apache.org/jira/browse/SOLR-9633 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shalin Shekhar Mangar > Fix For: master (7.0), 6.4 > > Attachments: SOLR-9633.patch, SOLR-9633.patch > > > SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on > memory usage. That helps with the query result cache but not with the filter > cache which defaults to FastLRUCache. This issue intends to add the same > feature to FastLRUCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[
https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666161#comment-15666161
]
Shalin Shekhar Mangar commented on SOLR-9633:
-
I put SOLR-9366 instead of SOLR-9633 in the commit message by mistake. The
commit messages are:
{code}
Commit 487b0976eb3e98b78ab492f4969a2aa0373b626f in lucene-solr's branch
refs/heads/master from Shalin Shekhar Mangar
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=487b097 ]
SOLR-9366: Limit memory consumed by FastLRUCache with a new 'maxRamMB' config
parameter
{code}
{code}
Commit 53dad4f0d1c8f563810626ef9ab470fb3e5d1da9 in lucene-solr's branch
refs/heads/branch_6x from Shalin Shekhar Mangar
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=53dad4f ]
SOLR-9366: Limit memory consumed by FastLRUCache with a new 'maxRamMB' config
parameter
(cherry picked from commit 487b097)
{code}
> Limit FastLRUCache by RAM Usage
> ---
>
> Key: SOLR-9633
> URL: https://issues.apache.org/jira/browse/SOLR-9633
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Shalin Shekhar Mangar
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9633.patch, SOLR-9633.patch
>
>
> SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on
> memory usage. That helps with the query result cache but not with the filter
> cache which defaults to FastLRUCache. This issue intends to add the same
> feature to FastLRUCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[ https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654731#comment-15654731 ] Michael Sun commented on SOLR-9633: --- The patch looks good to me. One suggestion is to add comments that RAM limit feature only works well when the value in cache is an Accountable, such as filter cache whose value is a DocSet, which is an Accountable. In case user want to use FastLRUCache for other cache, user need to make sure the type of value is Accountable. Another minor general suggestion is to use PriorityQueue in markAndSweepByRamSize(), instead of Collection.sort(). It probably doesn't make big difference since most cache size is less than 1000 though. > Limit FastLRUCache by RAM Usage > --- > > Key: SOLR-9633 > URL: https://issues.apache.org/jira/browse/SOLR-9633 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shalin Shekhar Mangar > Fix For: master (7.0), 6.4 > > Attachments: SOLR-9633.patch > > > SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on > memory usage. That helps with the query result cache but not with the filter > cache which defaults to FastLRUCache. This issue intends to add the same > feature to FastLRUCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[ https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645182#comment-15645182 ] Yonik Seeley commented on SOLR-9633: OK, thanks for the explanation. That's clearer after I apply the patch too. Things look good... I think the only minor change I'd make is to wrap the other update to ramBytes in the put() method with the "ramUpperWatermark != Long.MAX_VALUE" check as you did in other places. > Limit FastLRUCache by RAM Usage > --- > > Key: SOLR-9633 > URL: https://issues.apache.org/jira/browse/SOLR-9633 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shalin Shekhar Mangar > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9633.patch > > > SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on > memory usage. That helps with the query result cache but not with the filter > cache which defaults to FastLRUCache. This issue intends to add the same > feature to FastLRUCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[ https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15643091#comment-15643091 ] Shalin Shekhar Mangar commented on SOLR-9633: - The size based eviction logic in markAndSweep method is moved to markAndSweepByCacheSize -- there are no logic changes when only item size is used for eviction. But if maxRamMB is configured then we use the markAndSweepByRamSize method which evicts by making two passes over the cache and a sort. After these changes, the FastLRUCache behaves differently than LRUCache. Instead of supporting both size and maxRamMB and evicting based on both limits, the FastLRUCache supports either size or maxRamMB but not both. This is also because I couldn't figure out a way to evict by ram as efficiently as we do today by size. We could support both size and ram together by evicting by size first and then fall back to the expensive two pass eviction based on ram. > Limit FastLRUCache by RAM Usage > --- > > Key: SOLR-9633 > URL: https://issues.apache.org/jira/browse/SOLR-9633 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shalin Shekhar Mangar > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9633.patch > > > SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on > memory usage. That helps with the query result cache but not with the filter > cache which defaults to FastLRUCache. This issue intends to add the same > feature to FastLRUCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[ https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15642418#comment-15642418 ] Yonik Seeley commented on SOLR-9633: Can you explain what logic changes are in markAndSweepByCacheSize()? It's hard to tell looking at the diff (lots of indent changes... not sure about other changes). > Limit FastLRUCache by RAM Usage > --- > > Key: SOLR-9633 > URL: https://issues.apache.org/jira/browse/SOLR-9633 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shalin Shekhar Mangar > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9633.patch > > > SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on > memory usage. That helps with the query result cache but not with the filter > cache which defaults to FastLRUCache. This issue intends to add the same > feature to FastLRUCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[ https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15642398#comment-15642398 ] Yonik Seeley commented on SOLR-9633: Taking a look... > Limit FastLRUCache by RAM Usage > --- > > Key: SOLR-9633 > URL: https://issues.apache.org/jira/browse/SOLR-9633 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shalin Shekhar Mangar > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9633.patch > > > SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on > memory usage. That helps with the query result cache but not with the filter > cache which defaults to FastLRUCache. This issue intends to add the same > feature to FastLRUCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[ https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636910#comment-15636910 ] Shalin Shekhar Mangar commented on SOLR-9633: - [~ysee...@gmail.com] - do you mind taking a look at this? Is the simple ram size based eviction logic fine or do we need to implement something fancier as we have for size based eviction? > Limit FastLRUCache by RAM Usage > --- > > Key: SOLR-9633 > URL: https://issues.apache.org/jira/browse/SOLR-9633 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shalin Shekhar Mangar > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9633.patch > > > SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on > memory usage. That helps with the query result cache but not with the filter > cache which defaults to FastLRUCache. This issue intends to add the same > feature to FastLRUCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9633) Limit FastLRUCache by RAM Usage
[ https://issues.apache.org/jira/browse/SOLR-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576099#comment-15576099 ] Shalin Shekhar Mangar commented on SOLR-9633: - FYI [~ysee...@gmail.com] > Limit FastLRUCache by RAM Usage > --- > > Key: SOLR-9633 > URL: https://issues.apache.org/jira/browse/SOLR-9633 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shalin Shekhar Mangar > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9633.patch > > > SOLR-7372 added a maxRamMB parameter to LRUCache to evict items based on > memory usage. That helps with the query result cache but not with the filter > cache which defaults to FastLRUCache. This issue intends to add the same > feature to FastLRUCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
