[
https://issues.apache.org/jira/browse/SOLR-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030952#comment-17030952
]
Erick Erickson commented on SOLR-5146:
--------------------------------------
[~murblanc] That's certainly one issue. Even if efficiently getting a leader
for a completely unloaded shard is solved, the question of how to keep the core
in sync is a sticky wicket. Say even one replica of a shard is unloaded and it
gets loaded. How is the core synched before doing anything? If replicas are
coming and going all the time, do we wind up doing full synchronizations
(assuming the leader problem is solved)? In the case of, say, 200G indexes for
a given replica, that's very expensive.
Core loading from a cold start is a very heavyweight operation. It may be that
we need some intermediate state where we can free up lots of resources but keep
the core kind of loaded, mostly so it could be waked up nearly instantly, say
the equivalent of opening a new searcher.
Leader election is really all about insuring that the index is up to date. So
I've wondered about a state for a replica that's "index only" rather than
unloaded, the idea is that that way it's always up to date and can (almost)
instantly assume leadership, but doesn't consume the heavier-weight resources.
Then it could be brought online without having to sync from the leader. And
then "somehow" combine it with autoscaling-like functionality, when they query
rate exceeded X, bring another replica from index-only to serving searchers.
That'd take untangling what's necessary for indexing and what's necessary for
searching so they were relatively independent.
But I'll leave that for David to struggle with...
> Figure out what it would take for lazily-loaded cores to play nice with
> SolrCloud
> ---------------------------------------------------------------------------------
>
> Key: SOLR-5146
> URL: https://issues.apache.org/jira/browse/SOLR-5146
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Affects Versions: 4.5, 6.0
> Reporter: Erick Erickson
> Assignee: David Smiley
> Priority: Major
>
> The whole lazy-load core thing was implemented with non-SolrCloud use-cases
> in mind. There are several user-list threads that ask about using lazy cores
> with SolrCloud, especially in multi-tenant use-cases.
> This is a marker JIRA to investigate what it would take to make lazy-load
> cores play nice with SolrCloud. It's especially interesting how this all
> works with shards, replicas, leader election, recovery, etc.
> NOTE: This is pretty much totally unexplored territory. It may be that a few
> trivial modifications are all that's needed. OTOH, It may be that we'd have
> to rip apart SolrCloud to handle this case. Until someone dives into the
> code, we don't know.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
