[
https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156290#comment-15156290
]
Gus Heck commented on SOLR-8349:
--------------------------------
That sounds like a very cool way to store the source data, but this issue isn't
about access to the blob/file itself. It's about access to the in memory java
object created from the file/blob/whatever, so each core does not have to hold
it's own identical copy in RAM. In our case we had a csv file that was
translated into a HashMap (for fast access *during* query processing) that had
>900MB of memory usage. When we fired up 40 cores (20 collections each with a
main core and one replica) that used this component RAM usage was through the
roof with no data in the index yet.
A similar situation is faced by Analyzers that have dictionaries etc (see
SOLR-3443). The current patch no longer supports a solution to that issue, as
support for Lucene layer stuff has been deferred per discussion above.
It is interesting to see that there is an implementation of the ref counting to
use as a model, and your notes about hooks on the component will be useful too.
That may come in handy if/when a refresh mechanism is desired.
> Allow sharing of large in memory data structures across cores
> -------------------------------------------------------------
>
> Key: SOLR-8349
> URL: https://issues.apache.org/jira/browse/SOLR-8349
> Project: Solr
> Issue Type: Improvement
> Components: Server
> Affects Versions: 5.3
> Reporter: Gus Heck
> Attachments: SOLR-8349.patch, SOLR-8349.patch
>
>
> In some cases search components or analysis classes may utilize a large
> dictionary or other in-memory structure. When multiple cores are loaded with
> identical configurations utilizing this large in memory structure, each core
> holds it's own copy in memory. This has been noted in the past and a specific
> case reported in SOLR-3443. This patch provides a generalized capability, and
> if accepted, this capability will then be used to fix SOLR-3443.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
