[
https://issues.apache.org/jira/browse/SOLR-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136004#comment-14136004
]
Erick Erickson edited comment on SOLR-6491 at 9/16/14 7:25 PM:
---------------------------------------------------------------
bq: A node may be preferred for sometime and after sometime it can change
True. That's where the SOLR-6513 comes in. Remember also that this is a
_preference_, not an absolute map.
bq: After a new leader election happens overseer does a quick check on all the
nodes where the shard has replicas
One situation we're talking about here is one in which the pathology has
already happened. For illustration, consider the following:
I have 10 shards and 10 nodes. I have 100 replicas, i.e. each node has one
replica for every shard.
All the nodes are shut down.
I start 1 node. All 10 leaders are elected on that node.
Now I bring up my other 9 machines.
No leader election goes on here as the other 9 nodes come up. So "After a new
leader election happens overseer does a quick check on all the nodes where the
shard has replicas, and find out if there is a more suitable node for that
shard in the cluster" doesn't apply since there's no leader election going on.
I suppose we could just fire a "balance leaders" command, but that kind of
pre-supposes a symmetric cluster.
I'm reluctant to try to do everything automatically. Consider an asymmetric
cluster, both in terms of hardware and collection characteristics. You have
machines of varying power hosting (perhaps many) collections with different
usage patterns. I don't see how to replace the operations folks' knowledge of
all this with an auto-assignment process; the "and find out if there is a more
suitable node" above. Suitable by what criteria? To do this really well, we'd
need to have a language to define capabilities for both collections and nodes,
and a heuristic to factor all that in auto-leader election.
If we do take a stab at figuring this all out automatically, it seems that the
preferredLeader mechanism could be the way this is realized; auto-assign the
roles, then trigger the re-election process. With room for the ops people to
tweak it as they see fit.
Now all _that_ said, maybe I'm over-thinking the problem. It might be "good
enough" to just not bother with the preferredLeader role and have a "rebalance
leaders now" command. And trust the operations people to distribute their
collections appropriately. By "appropriately", I'm thinking of assigning the
collections across the cluster such that the leaders will be acceptably
distributed if we just balance them within a collection.
BTW, I think the idea of more intelligently electing leaders is a good one, and
this in no way means we shouldn't do that too. I'm concerned more intelligence
there is only part of the total solution though.
was (Author: erickerickson):
bq: A node may be preferred for sometime and after sometime it can change
True. That's where the SOLR-6513 comes in. Remember also that this is a
_preference_, not an absolute map.
bq: After a new leader election happens overseer does a quick check on all the
nodes where the shard has replicas
One situation we're talking about here is one in which the pathology has
already happened. For illustration, consider the following:
I have 10 shards and 10 nodes. I have 100 replicas, i.e. each node has one
replica for every shard.
All the nodes are shut down.
I start 1 node. All 10 leaders are elected on that node.
Now I bring up my other 9 machines.
No leader election goes on here as the other 9 nodes come up. So "After a new
leader election happens overseer does a quick check on all the nodes where the
shard has replicas, and find out if there is a more suitable node for that
shard in the cluster" doesn't apply since there's no leader election going on.
I suppose we could just fire a "balance leaders" command, but that kind of
pre-supposes a symmetric cluster.
I'm reluctant to try to do everything automatically. Consider an asymmetric
cluster, both in terms of hardware and collection characteristics. You have
machines of varying power hosting (perhaps many) collections with different
usage patterns. I don't see how to replace the operations folks' knowledge of
all this with an auto-assignment process; the "and find out if there is a more
suitable node" above. Suitable by what criteria? To do this really well, we'd
need to have a language to define capabilities for both collections and nodes,
and a heuristic to factor all that in auto-leader election.
If we do take a stab at figuring this all out automatically, it seems that the
preferredLeader mechanism could be the way this is realized; auto-assign the
roles, then trigger the re-election process. With room for the ops people to
tweak it as they see fit.
Now all _that_ said, maybe I'm over-thinking the problem. It might be "good
enough" to just not bother with the preferredLeader role and have a "rebalance
leaders now" command. And trust the operations people to distribute their
collections appropriately. By "appropriately", I'm thinking of assigning the
collections across the cluster such that the leaders will be acceptably
distributed if we just balance them within a collection.
> Umbrella JIRA for managing the leader assignments
> -------------------------------------------------
>
> Key: SOLR-6491
> URL: https://issues.apache.org/jira/browse/SOLR-6491
> Project: Solr
> Issue Type: Improvement
> Affects Versions: 4.11, 5.0
> Reporter: Erick Erickson
> Assignee: Erick Erickson
>
> Leaders can currently get out of balance due to the sequence of how nodes are
> brought up in a cluster. For very good reasons shard leadership cannot be
> permanently assigned.
> However, it seems reasonable that a sys admin could optionally specify that a
> particular node be the _preferred_ leader for a particular collection/shard.
> During leader election, preference would be given to any node so marked when
> electing any leader.
> So the proposal here is to add another role for preferredLeader to the
> collections API, something like
> ADDROLE?role=preferredLeader&collection=collection_name&shard=shardId
> Second, it would be good to have a new collections API call like
> ELECTPREFERREDLEADERS?collection=collection_name
> (I really hate that name so far, but you see the idea). That command would
> (asynchronously?) make an attempt to transfer leadership for each shard in a
> collection to the leader labeled as the preferred leader by the new ADDROLE
> role.
> I'm going to start working on this, any suggestions welcome!
> This will subsume several other JIRAs, I'll link them momentarily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
