[
https://issues.apache.org/jira/browse/SOLR-8906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15899812#comment-15899812
]
Erick Erickson commented on SOLR-8906:
--------------------------------------
bq: why do we want to unload a core?
Well, I know of at least one situation where people have implemented
auto-scaling that can do things like:
> split an index in the background. More generally maintain indexes of size no
> larger than X. Then move one or more of the splits "someplace else".
> move a core's index to SSD while it's hot.
> re-partition user's data based on some heuristics without downtime.
> any situation where a core is manipulated outside Solr and still needs to
> service requests in the mean time.
So I do wonder if we can stand the question on its head. Rather than think of
it as the transient cores being an afterthought, what if we move _all_ core
management to a plugin? NOTE: this is _really_ fuzzy ATM, just askin'.
Then we wouldn't have the distinction _in solr_ of "transient", "lazy loading",
"regular" deeply embedded in the CoreContainer & etc. code. Even in the case
where we open/close the heavyweight objects rather than load/unload cores, we
still have to maintain lists of what cores have searchers already open and the
like, similar to what happens in transient cores. Does it make any sense to
think of moving all core management to a (suitably modified) transient core
plugin? Then the default implementation we provide would just manage the
heavyweight objects rather than load/unload cores and others could do as they
wished.
Going forward, when everything is SolrCloud, there would be a degenerate case
of leader-only collections that could essentially be treated as we do the
current standalone code I'd guess.
bq: lazy cores itself is a vestige a of the old master-slave model
Not at all sure I agree. Even when SolrCloud rules the world, there'll always
be edge cases where some organization pushes the limit. I don't want to keep
Solr from evolving just to accommodate these edge cases, but I also don't want
to prematurely decide _for_ them that "we can do it better". 'cause we can't in
situation N+1.
Oh, and let's keep a distinction between "lazy" and "transient" cores. "lazy"
just means it isn't loaded until it's called for, it can be permanent after
that. "transient" is the whole cache-and-load/unload-when-needed bit. Don't
quite know how those will reconcile going forward, but the idea of
opening/closing heavyweight objects is still "lazy" cores in some sense.
> Make transient core cache pluggable.
> ------------------------------------
>
> Key: SOLR-8906
> URL: https://issues.apache.org/jira/browse/SOLR-8906
> Project: Solr
> Issue Type: Improvement
> Reporter: Erick Erickson
> Assignee: Erick Erickson
>
> The current Lazy Core stuff is pretty deeply intertwined in CoreContainer.
> Adding and removing active cores is based on a simple LRU mechanism, but
> keeping the right cores in the right internal structures involves a lot of
> attention to locking various objects to update internal structures. This
> makes it difficult/dangerous to use any other caching algorithms.
> Any single age-out algorithm will have non-optimal access patterns, so making
> this pluggable would allow better algorithms to be substituted in those cases.
> If we ever extend transient cores to SolrCloud, we need to have load/unload
> decisions that are cloud-aware rather then entirely local so in that sense
> this is would lay some groundwork if we ever want to go there.
> So I'm going to try to hack together a PoC. Any ideas on the most sensible
> pattern for this gratefully received.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
