Boyi Dai created SOLR-18601:
-------------------------------
Summary: A failed MIGRATE leaves the target leader buffering
updates, and retrying the MIGRATE fails
Key: SOLR-18601
URL: https://issues.apache.org/jira/browse/SOLR-18601
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Boyi Dai
MigrateCmd.migrateKey() first asks the target shard leader to buffer updates
(REQUESTBUFFERUPDATES), then adds a routing rule to the source shard that
forwards updates for the split key to the target. Only at the end, after
creating a temporary collection, splitting the source index into it, adding a
replica of it on the target node and merging that into the target, does it ask
the target leader to apply the buffered updates (REQUESTAPPLYUPDATES).
If any step in between fails (e.g. the routing rule doesn't show up in time, or
the temporary collection can't be created), the exception propagates and
nothing undoes the earlier steps:
* The target leader's update log stays in the BUFFERING state. Updates the
source forwards under the routing rule are written to the buffer and never
become searchable on the target.
* The routing rule stays on the source shard until it expires (forward.timeout,
10 minutes by default).
* Retrying the same MIGRATE fails at its first step: REQUESTBUFFERUPDATES
requires the update log to be ACTIVE (CoreAdminOperation), so the retry fails
with "MIGRATE failed to request node to buffer updates". Getting out of this
needs a manual core admin call (REQUESTAPPLYUPDATES) on the target leader.
This was found through code analysis, and I reproduced it with a test: a
MIGRATE that fails while creating its temporary collection leaves the target
leader in BUFFERING, synchronously and asynchronously, and a retry then fails
as above.
*Steps to reproduce:*
# Start a MIGRATE of a split key from collection S to collection T, and have it
fail after the target leader started buffering (in the test: an alias refers to
a leftover temporary collection split_shard1_temp_shard1, so MIGRATE can't
delete it and then fails to create it).
# Check the update log state of T's leader: it is still BUFFERING.
# Retry the MIGRATE once the cause is gone.
# Result: the retry fails with "MIGRATE failed to request node to buffer
updates".
*Expected:* a failed MIGRATE should undo its routing rule and the buffering on
the target, so that the target behaves as before and a retry works.
*Proposed fix:* when a step fails before the target leader is asked to apply
its buffered updates, remove the routing rule from the source shard and drop
the target leader's buffered updates. Dropping (rather than applying) puts the
target back in its pre-MIGRATE state; the source shard still has all the
forwarded documents, since forwarding happens in addition to indexing them on
the source. There is no core admin action to drop buffered updates yet
(UpdateLog.dropBufferedUpdates() exists but is unused), so the fix adds an
internal REQUESTDROPBUFFEREDUPDATES action alongside REQUESTBUFFERUPDATES and
REQUESTAPPLYUPDATES. I have a patch with a test and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
