[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-8349: --- Fix Version/s: (was: 6.0) master (7.0) Manually correcting fixVersion per Step #S5 of LUCENE-7271 > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Sub-task > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Fix For: 6.1, master (7.0) > > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: (was: SOLR_8349.patch) > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Sub-task > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: SOLR-8349.patch Patch vs origin/master. Only conflict was a change to use signature for an updated library version in code I am replacing anyway. > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Sub-task > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR_8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-8349: - Issue Type: Sub-task (was: Improvement) Parent: SOLR-8773 > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Sub-task > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, SOLR_8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: SOLR_8349.patch Unit tests, both cloud style and mock. Everything passed precommit before I updated... after update the changes to the spatial stuff broke my IDE and precommit, but there were no conflicts with my code so this should still be ok > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Improvement > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, SOLR_8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: SOLR-8349.patch Actual patch that supports multiple decodings. > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Improvement > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: SOLR-8349.patch Patch with support for multiple decodings of the same blob > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Improvement > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349.patch, SOLR-8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: (was: SOLR-8349_4.patch) > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Improvement > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: SOLR-8349.patch patch vs up to date master > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Improvement > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349.patch, SOLR-8349_4.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: SOLR-8349_4.patch > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Improvement > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch, > SOLR-8349_4.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-8349: - Attachment: SOLR-8349.patch [~gus_heck] untested patch . should be good enough for you to go ahead > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Improvement > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck >Assignee: Noble Paul > Attachments: SOLR-8349.patch, SOLR-8349.patch, SOLR-8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: SOLR-8349.patch Changes since original patch: # No interface added to lucene, and no support for lucene analyzers to use this feature at this time. (defer to subsequent enhancement) # Caching now done using a Guava cache. This simplifies code, but means that a core loading a given resource will now block the loading of other cores that also require that resource. # As suggested by Dave, the cache now uses weak references to the values avoiding the possibility of a ClassLoader memory leak. The SolrCore object now gets a hard reference to the object. This means that there is now some chance that a resource will be unloaded if the last core using it is unloaded and then loaded again. This differs from the previous code where it was guaranteed to not be unloaded. To guarantee an deterministic behavior we likely would need to use hard references in the cache as before, or add something along the lines of a reference counting scheme to know when the last core stopped using it. # To ensure that the SolrCore is given a reference to the loaded object, a ContainerResourceAware interface was added and is treated similarly to SolrCoreAware and ResourceLoaderAware. This allows components to give us the loading code before we give them the core, and thus we can coordinate the loading such that no there is always a hard reference to the resource (until the last core using it is unloaded). > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Improvement > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck > Attachments: SOLR-8349.patch, SOLR-8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8349) Allow sharing of large in memory data structures across cores
[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-8349: --- Attachment: SOLR-8349.patch Patch implementing general feature. Patch for SOLR-3443 to be attached there shortly. > Allow sharing of large in memory data structures across cores > - > > Key: SOLR-8349 > URL: https://issues.apache.org/jira/browse/SOLR-8349 > Project: Solr > Issue Type: Improvement > Components: Server >Affects Versions: 5.3 >Reporter: Gus Heck > Attachments: SOLR-8349.patch > > > In some cases search components or analysis classes may utilize a large > dictionary or other in-memory structure. When multiple cores are loaded with > identical configurations utilizing this large in memory structure, each core > holds it's own copy in memory. This has been noted in the past and a specific > case reported in SOLR-3443. This patch provides a generalized capability, and > if accepted, this capability will then be used to fix SOLR-3443. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
