Torsten Bøgh Köster created SOLR-10550:
------------------------------------------
Summary: Improve FileFloatSource eviction // reduce
FileFloatSource memory footprint
Key: SOLR-10550
URL: https://issues.apache.org/jira/browse/SOLR-10550
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: Server
Affects Versions: 6.5
Reporter: Torsten Bøgh Köster
Attachments: solr_filefloatsource.patch
As a follow up from {{SOLR-10506}} we found another possible memory leak in
Solr. The values generated from an {{ExternalFileField}} are cached in a static
cache inside the {{FileFloatSource}}. That cache caches both a {{IndexReader}}
and {{FileFloatSource}}s loaded using that {{IndexReader}}.
Cache eviction is left to the internally used {{WeakHashMap}} or a full
eviction can be triggered via url. We are dealing with large synonym files and
word lists stored in managed resources. Those are tied to the {{SolrCore}} as
described in {{SOLR-10506}}. We're also using {{ExternalFileField}}s whose
{{FileFloatSource}} are cached in said static cache. The {{FileFloatSource}}
hold strong (transitive) references to the {{SolrCore}} they have been created
for.
After a couple of collection reloads, the cache eviction mechanism of the
{{WeakHashMap}} gets activated pretty close to heap exhaustion. The patch
attached adds a mechanism to evict cache entries created in the context of a
{{SolrCore}} upon it's close using a close hook in the
{{ExternalFileFieldReloader}}. It furthermore adds a static cache reset method
for all entries bound to a given {{IndexReader}}. I'm not sure, if the added
cache resets are too aggressive or executed too often, I'd like to leave that
to the experts ;-)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
