[
https://issues.apache.org/jira/browse/SOLR-18601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18601:
----------------------------------
Labels: pull-request-available (was: )
> A failed MIGRATE leaves the target leader buffering updates, and retrying the
> MIGRATE fails
> -------------------------------------------------------------------------------------------
>
> Key: SOLR-18601
> URL: https://issues.apache.org/jira/browse/SOLR-18601
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Boyi Dai
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> MigrateCmd.migrateKey() first asks the target shard leader to buffer updates
> (REQUESTBUFFERUPDATES), then adds a routing rule to the source shard that
> forwards updates for the split key to the target. Only at the end, after
> creating a temporary collection, splitting the source index into it, adding a
> replica of it on the target node and merging that into the target, does it
> ask the target leader to apply the buffered updates (REQUESTAPPLYUPDATES).
> If any step in between fails (e.g. the routing rule doesn't show up in time,
> or the temporary collection can't be created), the exception propagates and
> nothing undoes the earlier steps:
> * The target leader's update log stays in the BUFFERING state. Updates the
> source forwards under the routing rule are written to the buffer and never
> become searchable on the target.
> * The routing rule stays on the source shard until it expires
> (forward.timeout, 10 minutes by default).
> * Retrying the same MIGRATE fails at its first step: REQUESTBUFFERUPDATES
> requires the update log to be ACTIVE (CoreAdminOperation), so the retry fails
> with "MIGRATE failed to request node to buffer updates". Getting out of this
> needs a manual core admin call (REQUESTAPPLYUPDATES) on the target leader.
> This was found through code analysis, and I reproduced it with a test: a
> MIGRATE that fails while creating its temporary collection leaves the target
> leader in BUFFERING, synchronously and asynchronously, and a retry then fails
> as above.
> *Steps to reproduce:*
> # Start a MIGRATE of a split key from collection S to collection T, and have
> it fail after the target leader started buffering (in the test: an alias
> refers to a leftover temporary collection split_shard1_temp_shard1, so
> MIGRATE can't delete it and then fails to create it).
> # Check the update log state of T's leader: it is still BUFFERING.
> # Retry the MIGRATE once the cause is gone.
> # Result: the retry fails with "MIGRATE failed to request node to buffer
> updates".
> *Expected:* a failed MIGRATE should undo its routing rule and the buffering
> on the target, so that the target behaves as before and a retry works.
> *Proposed fix:* when a step fails before the target leader is asked to apply
> its buffered updates, remove the routing rule from the source shard and drop
> the target leader's buffered updates. Dropping (rather than applying) puts
> the target back in its pre-MIGRATE state; the source shard still has all the
> forwarded documents, since forwarding happens in addition to indexing them on
> the source. There is no core admin action to drop buffered updates yet
> (UpdateLog.dropBufferedUpdates() exists but is unused), so the fix adds an
> internal REQUESTDROPBUFFEREDUPDATES action alongside REQUESTBUFFERUPDATES and
> REQUESTAPPLYUPDATES. I have a patch with a test and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
