[
https://issues.apache.org/jira/browse/SOLR-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-5468:
---------------------------------
Attachment: SOLR-5468.patch
Here's an updated patch that may prove useful for client applications that wish
to be notified (via responseHeader) that a desired level of replication was or
was not achieved for an update request.
Since this is not something that all client applications would need, it must be
explicitly activated by passing the min_rf=N where N>1, e.g.
.../update?min_rf=2 will activate this feature. The actual value of the min_rf
is only meaningful to the client application and server code will simply report
what level was achieved. Originally I was thinking that
DistributedUpdateProcessor only needed to block until the desired level was
achieved but it turns out that it blocks until all replicas are tried anyway.
The responseHeader would then include the minimum achieved replication factor
for that request, such as:
<response>
<lst name="responseHeader"><int name="rf">2</int><int name="min_rf">2</int><int
name="status">0</int><int name="QTime">127</int></lst>
</response>
To be clear, it is the client application's responsibility to read this
information from the responseHeader and act accordingly. To help out with this,
esp if using CloudSolrServer, client applications call call
CloudSolrServer.getMinAchievedReplicationFactor, which will return the minimum
achieved replication factor across all shards for a request. Of course, for
single doc update requests, there will only be one shard involved.
Admittedly, this code might look a bit cumbersome just to produce a simple
count, see: DistributedUpdateProcess.RequestReplicationTracker inner class.
This approach is needed to support batch update requests, including those that
don't come in from directUpdates from CloudSolrServer (hence the need for the
outer Map to keep track of replication outcomes per shard).
Wanted to solicit some feedback here before committing but the overall impact
to existing code is minimal as this optional feature must be activated
intentionally by the client and only introduces minimal overhead.
> Option to enforce a majority quorum approach to accepting updates in SolrCloud
> ------------------------------------------------------------------------------
>
> Key: SOLR-5468
> URL: https://issues.apache.org/jira/browse/SOLR-5468
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
> Affects Versions: 4.5
> Environment: All
> Reporter: Timothy Potter
> Assignee: Timothy Potter
> Priority: Minor
> Attachments: SOLR-5468.patch, SOLR-5468.patch
>
>
> I've been thinking about how SolrCloud deals with write-availability using
> in-sync replica sets, in which writes will continue to be accepted so long as
> there is at least one healthy node per shard.
> For a little background (and to verify my understanding of the process is
> correct), SolrCloud only considers active/healthy replicas when acknowledging
> a write. Specifically, when a shard leader accepts an update request, it
> forwards the request to all active/healthy replicas and only considers the
> write successful if all active/healthy replicas ack the write. Any down /
> gone replicas are not considered and will sync up with the leader when they
> come back online using peer sync or snapshot replication. For instance, if a
> shard has 3 nodes, A, B, C with A being the current leader, then writes to
> the shard will continue to succeed even if B & C are down.
> The issue is that if a shard leader continues to accept updates even if it
> loses all of its replicas, then we have acknowledged updates on only 1 node.
> If that node, call it A, then fails and one of the previous replicas, call it
> B, comes back online before A does, then any writes that A accepted while the
> other replicas were offline are at risk to being lost.
> SolrCloud does provide a safe-guard mechanism for this problem with the
> leaderVoteWait setting, which puts any replicas that come back online before
> node A into a temporary wait state. If A comes back online within the wait
> period, then all is well as it will become the leader again and no writes
> will be lost. As a side note, sys admins definitely need to be made more
> aware of this situation as when I first encountered it in my cluster, I had
> no idea what it meant.
> My question is whether we want to consider an approach where SolrCloud will
> not accept writes unless there is a majority of replicas available to accept
> the write? For my example, under this approach, we wouldn't accept writes if
> both B&C failed, but would if only C did, leaving A & B online. Admittedly,
> this lowers the write-availability of the system, so may be something that
> should be tunable?
> From Mark M: Yeah, this is kind of like one of many little features that we
> have just not gotten to yet. I’ve always planned for a param that let’s you
> say how many replicas an update must be verified on before responding
> success. Seems to make sense to fail that type of request early if you notice
> there are not enough replicas up to satisfy the param to begin with.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
