[
https://issues.apache.org/jira/browse/SOLR-18601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18134585#comment-18134585
]
Boyi Dai commented on SOLR-18601:
---------------------------------
Update: the fix in the linked PR turned out to be wrong. It drops the target
leader's buffered updates and removes the whole routing rule on failure, but
MIGRATE doesn't own either of them: routing rules for the same route key merge
their ranges, and the buffer is shared by every forwarding stream, recovery and
SPLITSHARD. A review on the PR also showed that async MIGRATE doesn't abort
when a step fails. I've moved the PR to draft.
The bug itself still stands (a failed MIGRATE leaves the target leader
BUFFERING and the retry fails). A correct fix seems to need ownership of the
buffering and stricter failure handling, so I'd like input on the intended
contract for a failed MIGRATE before reworking it: should it roll back
(dropping only what it owns) or leave the target with possibly partial data
after applying the buffer?
> A failed MIGRATE leaves the target leader buffering updates, and retrying the
> MIGRATE fails
> -------------------------------------------------------------------------------------------
>
> Key: SOLR-18601
> URL: https://issues.apache.org/jira/browse/SOLR-18601
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Boyi Dai
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> MigrateCmd.migrateKey() first asks the target shard leader to buffer updates
> (REQUESTBUFFERUPDATES), then adds a routing rule to the source shard that
> forwards updates for the split key to the target. Only at the end, after
> creating a temporary collection, splitting the source index into it, adding a
> replica of it on the target node and merging that into the target, does it
> ask the target leader to apply the buffered updates (REQUESTAPPLYUPDATES).
> If any step in between fails (e.g. the routing rule doesn't show up in time,
> or the temporary collection can't be created), the exception propagates and
> nothing undoes the earlier steps:
> * The target leader's update log stays in the BUFFERING state. Updates the
> source forwards under the routing rule are written to the buffer and never
> become searchable on the target.
> * The routing rule stays on the source shard until it expires
> (forward.timeout, 10 minutes by default).
> * Retrying the same MIGRATE fails at its first step: REQUESTBUFFERUPDATES
> requires the update log to be ACTIVE (CoreAdminOperation), so the retry fails
> with "MIGRATE failed to request node to buffer updates". Getting out of this
> needs a manual core admin call (REQUESTAPPLYUPDATES) on the target leader.
> This was found through code analysis, and I reproduced it with a test: a
> MIGRATE that fails while creating its temporary collection leaves the target
> leader in BUFFERING, synchronously and asynchronously, and a retry then fails
> as above.
> *Steps to reproduce:*
> # Start a MIGRATE of a split key from collection S to collection T, and have
> it fail after the target leader started buffering (in the test: an alias
> refers to a leftover temporary collection split_shard1_temp_shard1, so
> MIGRATE can't delete it and then fails to create it).
> # Check the update log state of T's leader: it is still BUFFERING.
> # Retry the MIGRATE once the cause is gone.
> # Result: the retry fails with "MIGRATE failed to request node to buffer
> updates".
> *Expected:* a failed MIGRATE should undo its routing rule and the buffering
> on the target, so that the target behaves as before and a retry works.
> *Proposed fix:* when a step fails before the target leader is asked to apply
> its buffered updates, remove the routing rule from the source shard and drop
> the target leader's buffered updates. Dropping (rather than applying) puts
> the target back in its pre-MIGRATE state; the source shard still has all the
> forwarded documents, since forwarding happens in addition to indexing them on
> the source. There is no core admin action to drop buffered updates yet
> (UpdateLog.dropBufferedUpdates() exists but is unused), so the fix adds an
> internal REQUESTDROPBUFFEREDUPDATES action alongside REQUESTBUFFERUPDATES and
> REQUESTAPPLYUPDATES. I have a patch with a test and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
