[
https://issues.apache.org/jira/browse/SOLR-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832469#comment-15832469
]
Shawn Heisey commented on SOLR-10015:
-------------------------------------
I may be showing my ignorance here, but I have a question.
Would this prune individual items from within a larger field cache entry, or
prune an entire set of items at once? The latter sounds OK, but the former
seems like a very bad idea. If one of those large entries in the cache were to
have a percentage of its elements removed by GC, I think that search results
would be incorrect.
I don't know enough about Lucene internals to be able to answer my question by
looking at the patch.
Assuming my worry above is unfounded, I think this is only likely to delay OOM
instead of preventing it, and even if it did prevent it, a low memory situation
would probably involve constant eviction (by GC) and recreation (by queries) of
FC entries, which I would expect to have strong performance implications.
> Remove strong reference to Field Cache key (optional) so that GC can release
> some Field Cache entries when Solr is under memory pressure
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-10015
> URL: https://issues.apache.org/jira/browse/SOLR-10015
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Michael Sun
> Attachments: SOLR-10015-prototype.patch
>
>
> In current Field Cache (FC) implementation, a WeakHashMap is used, supposedly
> designed to allow GC to release some Field Cache entries when Solr is under
> memory pressure.
> However, in practice, FC entry releasing seldom happens. Even worse, sometime
> Solr goes OOM and heap dump shows large amount of memory is actually used by
> FC. It's a sign that GC is not able to release FC entries even WeakHashMap is
> used.
> The reason is that FC is using SegmentCoreReaders as the key to the
> WeakHashMap. However, SegmentCoreReaders is usually strong referenced by
> SegmentReader. A strong reference would prevent GC to release the key and
> therefore the value. Therefore GC can't release entries in FC's WeakHashMap.
> The JIRA is to propose a solution to remove the strong reference mentioned
> above so that GC can release FC entries to avoid long GC pause or OOM. It
> needs to be optional because this change is a tradeoff, trading more CPU
> cycles for low memory footage. User can make final decision depending on
> their use cases.
> The prototype attached use a combination of directory name and segment name
> as key to the WeakHashMap, replacing the SegmentCoreReaders. Without change,
> Solr doesn't release any FC entries after a GC is manually triggered. With
> the change, FC entries are usually released after GC.
> However, I am not sure if it's the best way to solve this problem. Any
> suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
