[jira] [Commented] (SOLR-11882) SolrMetric registries retain references to SolrCores when closed
[ https://issues.apache.org/jira/browse/SOLR-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337104#comment-16337104 ] Eros Taborelli commented on SOLR-11882: --- [~erickerickson] yes, that is what we see. > SolrMetric registries retain references to SolrCores when closed > > > Key: SOLR-11882 > URL: https://issues.apache.org/jira/browse/SOLR-11882 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Server >Affects Versions: 7.1 >Reporter: Eros Taborelli >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-11882.patch, SOLR-11882.patch, create-cores.zip, > solr-dump-full_Leak_Suspects.zip, solr.config.zip > > > *Description:* > Our setup involves using a lot of small cores (possibly hundred thousand), > but working only on a few of them at any given time. > We already followed all recommendations in this guide: > [https://wiki.apache.org/solr/LotsOfCores] > We noticed that after creating/loading around 1000-2000 empty cores, with no > documents inside, the heap consumption went through the roof despite having > set transientCacheSize to only 64 (heap size set to 12G). > All cores are correctly set to loadOnStartup=false and transient=true, and we > have verified via logs that the cores in excess are actually being closed. > However, a reference remains in the > org.apache.solr.metrics.SolrMetricManager#registries that is never removed > until a core if fully unloaded. > Restarting the JVM loads all cores in the admin UI, but doesn't populate the > ConcurrentHashMap until a core is actually fully loaded. > I reproduced the issue on a smaller scale (transientCacheSize = 5, heap size > = 512m) and made a report (attached) using eclipse MAT. > *Desired outcome:* > When a transient core is closed, the references in the SolrMetricManager > should be removed, in the same fashion the reporters for the core are also > closed and removed. > In alternative, a unloadOnClose=true|false flag could be implemented to fully > unload a transient core when closed due to the cache size. > *Note:* > The documentation mentions everywhere that the unused cores will be unloaded, > but it's misleading as the cores are never fully unloaded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11882) SolrMetric registries cause huge memory consumption with lots of transient cores
[
https://issues.apache.org/jira/browse/SOLR-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335488#comment-16335488
]
Eros Taborelli commented on SOLR-11882:
---
[~erickerickson] I also uploaded our configuration and the test script used to
create multiple cores.
If you want to test it, you'll need to add -Dfts.solr.indexdatadir=
to your solr.in.sh.
Example:
{noformat}
SOLR_OPTS="$SOLR_OPTS -Dfts.solr.indexdatadir=/tmp/solrdata{noformat}
Then, run ./create-cores.sh \{1..1000}
Let me know if you need more details.
> SolrMetric registries cause huge memory consumption with lots of transient
> cores
>
>
> Key: SOLR-11882
> URL: https://issues.apache.org/jira/browse/SOLR-11882
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Server
>Affects Versions: 7.1
>Reporter: Eros Taborelli
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-11882.patch, create-cores.zip,
> solr-dump-full_Leak_Suspects.zip, solr.config.zip
>
>
> *Description:*
> Our setup involves using a lot of small cores (possibly hundred thousand),
> but working only on a few of them at any given time.
> We already followed all recommendations in this guide:
> [https://wiki.apache.org/solr/LotsOfCores]
> We noticed that after creating/loading around 1000-2000 empty cores, with no
> documents inside, the heap consumption went through the roof despite having
> set transientCacheSize to only 64 (heap size set to 12G).
> All cores are correctly set to loadOnStartup=false and transient=true, and we
> have verified via logs that the cores in excess are actually being closed.
> However, a reference remains in the
> org.apache.solr.metrics.SolrMetricManager#registries that is never removed
> until a core if fully unloaded.
> Restarting the JVM loads all cores in the admin UI, but doesn't populate the
> ConcurrentHashMap until a core is actually fully loaded.
> I reproduced the issue on a smaller scale (transientCacheSize = 5, heap size
> = 512m) and made a report (attached) using eclipse MAT.
> *Desired outcome:*
> When a transient core is closed, the references in the SolrMetricManager
> should be removed, in the same fashion the reporters for the core are also
> closed and removed.
> In alternative, a unloadOnClose=true|false flag could be implemented to fully
> unload a transient core when closed due to the cache size.
> *Note:*
> The documentation mentions everywhere that the unused cores will be unloaded,
> but it's misleading as the cores are never fully unloaded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11882) SolrMetric registries cause huge memory consumption with lots of transient cores
[ https://issues.apache.org/jira/browse/SOLR-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eros Taborelli updated SOLR-11882: -- Attachment: create-cores.zip > SolrMetric registries cause huge memory consumption with lots of transient > cores > > > Key: SOLR-11882 > URL: https://issues.apache.org/jira/browse/SOLR-11882 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Server >Affects Versions: 7.1 >Reporter: Eros Taborelli >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-11882.patch, create-cores.zip, > solr-dump-full_Leak_Suspects.zip, solr.config.zip > > > *Description:* > Our setup involves using a lot of small cores (possibly hundred thousand), > but working only on a few of them at any given time. > We already followed all recommendations in this guide: > [https://wiki.apache.org/solr/LotsOfCores] > We noticed that after creating/loading around 1000-2000 empty cores, with no > documents inside, the heap consumption went through the roof despite having > set transientCacheSize to only 64 (heap size set to 12G). > All cores are correctly set to loadOnStartup=false and transient=true, and we > have verified via logs that the cores in excess are actually being closed. > However, a reference remains in the > org.apache.solr.metrics.SolrMetricManager#registries that is never removed > until a core if fully unloaded. > Restarting the JVM loads all cores in the admin UI, but doesn't populate the > ConcurrentHashMap until a core is actually fully loaded. > I reproduced the issue on a smaller scale (transientCacheSize = 5, heap size > = 512m) and made a report (attached) using eclipse MAT. > *Desired outcome:* > When a transient core is closed, the references in the SolrMetricManager > should be removed, in the same fashion the reporters for the core are also > closed and removed. > In alternative, a unloadOnClose=true|false flag could be implemented to fully > unload a transient core when closed due to the cache size. > *Note:* > The documentation mentions everywhere that the unused cores will be unloaded, > but it's misleading as the cores are never fully unloaded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11882) SolrMetric registries cause huge memory consumption with lots of transient cores
[ https://issues.apache.org/jira/browse/SOLR-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eros Taborelli updated SOLR-11882: -- Attachment: solr.config.zip > SolrMetric registries cause huge memory consumption with lots of transient > cores > > > Key: SOLR-11882 > URL: https://issues.apache.org/jira/browse/SOLR-11882 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Server >Affects Versions: 7.1 >Reporter: Eros Taborelli >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-11882.patch, solr-dump-full_Leak_Suspects.zip, > solr.config.zip > > > *Description:* > Our setup involves using a lot of small cores (possibly hundred thousand), > but working only on a few of them at any given time. > We already followed all recommendations in this guide: > [https://wiki.apache.org/solr/LotsOfCores] > We noticed that after creating/loading around 1000-2000 empty cores, with no > documents inside, the heap consumption went through the roof despite having > set transientCacheSize to only 64 (heap size set to 12G). > All cores are correctly set to loadOnStartup=false and transient=true, and we > have verified via logs that the cores in excess are actually being closed. > However, a reference remains in the > org.apache.solr.metrics.SolrMetricManager#registries that is never removed > until a core if fully unloaded. > Restarting the JVM loads all cores in the admin UI, but doesn't populate the > ConcurrentHashMap until a core is actually fully loaded. > I reproduced the issue on a smaller scale (transientCacheSize = 5, heap size > = 512m) and made a report (attached) using eclipse MAT. > *Desired outcome:* > When a transient core is closed, the references in the SolrMetricManager > should be removed, in the same fashion the reporters for the core are also > closed and removed. > In alternative, a unloadOnClose=true|false flag could be implemented to fully > unload a transient core when closed due to the cache size. > *Note:* > The documentation mentions everywhere that the unused cores will be unloaded, > but it's misleading as the cores are never fully unloaded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-11882) SolrMetric registries cause huge memory consumption with lots of transient cores
Eros Taborelli created SOLR-11882: - Summary: SolrMetric registries cause huge memory consumption with lots of transient cores Key: SOLR-11882 URL: https://issues.apache.org/jira/browse/SOLR-11882 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: Server Affects Versions: 7.1 Reporter: Eros Taborelli Attachments: solr-dump-full_Leak_Suspects.zip *Description:* Our setup involves using a lot of small cores (possibly hundred thousand), but working only on a few of them at any given time. We already followed all recommendations in this guide: [https://wiki.apache.org/solr/LotsOfCores] We noticed that after creating/loading around 1000-2000 empty cores, with no documents inside, the heap consumption went through the roof despite having set transientCacheSize to only 64 (heap size set to 12G). All cores are correctly set to loadOnStartup=false and transient=true, and we have verified via logs that the cores in excess are actually being closed. However, a reference remains in the org.apache.solr.metrics.SolrMetricManager#registries that is never removed until a core if fully unloaded. Restarting the JVM loads all cores in the admin UI, but doesn't populate the ConcurrentHashMap until a core is actually fully loaded. I reproduced the issue on a smaller scale (transientCacheSize = 5, heap size = 512m) and made a report (attached) using eclipse MAT. *Desired outcome:* When a transient core is closed, the references in the SolrMetricManager should be removed, in the same fashion the reporters for the core are also closed and removed. In alternative, a unloadOnClose=true|false flag could be implemented to fully unload a transient core when closed due to the cache size. *Note:* The documentation mentions everywhere that the unused cores will be unloaded, but it's misleading as the cores are never fully unloaded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
