[
https://issues.apache.org/jira/browse/SOLR-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Miller resolved SOLR-5552.
-------------------------------
Resolution: Fixed
> Leader recovery process can select the wrong leader if all replicas for a
> shard are down and trying to recover as well as lose updates that should have
> been recovered.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-5552
> URL: https://issues.apache.org/jira/browse/SOLR-5552
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Timothy Potter
> Assignee: Mark Miller
> Priority: Critical
> Labels: leader, recovery
> Fix For: 5.0, 4.7, 4.6.1
>
> Attachments: SOLR-5552.patch, SOLR-5552.patch
>
>
> One particular issue that leads to out-of-sync shards, related to SOLR-4260
> Here's what I know so far, which admittedly isn't much:
> As cloud85 (replica before it crashed) is initializing, it enters the wait
> process in ShardLeaderElectionContext#waitForReplicasToComeUp; this is
> expected and a good thing.
> Some short amount of time in the future, cloud84 (leader before it crashed)
> begins initializing and gets to a point where it adds itself as a possible
> leader for the shard (by creating a znode under
> /collections/cloud/leaders_elect/shard1/election), which leads to cloud85
> being able to return from waitForReplicasToComeUp and try to determine who
> should be the leader.
> cloud85 then tries to run the SyncStrategy, which can never work because in
> this scenario the Jetty HTTP listener is not active yet on either node, so
> all replication work that uses HTTP requests fails on both nodes ... PeerSync
> treats these failures as indicators that the other replicas in the shard are
> unavailable (or whatever) and assumes success. Here's the log message:
> 2013-12-11 11:43:25,936 [coreLoadExecutor-3-thread-1] WARN
> solr.update.PeerSync - PeerSync: core=cloud_shard1_replica1
> url=http://cloud85:8985/solr couldn't connect to
> http://cloud84:8984/solr/cloud_shard1_replica2/, counting as success
> The Jetty HTTP listener doesn't start accepting connections until long after
> this process has completed and already selected the wrong leader.
> From what I can see, we seem to have a leader recovery process that is based
> partly on HTTP requests to the other nodes, but the HTTP listener on those
> nodes isn't active yet. We need a leader recovery process that doesn't rely
> on HTTP requests. Perhaps, leader recovery for a shard w/o a current leader
> may need to work differently than leader election in a shard that has
> replicas that can respond to HTTP requests? All of what I'm seeing makes
> perfect sense for leader election when there are active replicas and the
> current leader fails.
> All this aside, I'm not asserting that this is the only cause for the
> out-of-sync issues reported in this ticket, but it definitely seems like it
> could happen in a real cluster.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
