[
https://issues.apache.org/jira/browse/SOLR-18584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18134685#comment-18134685
]
Boyi Dai commented on SOLR-18584:
---------------------------------
The PR is a draft for now. Three approaches each had a reproducible problem:
deleting on failure can itself fail, retrying activation leaves the collection
under-replicated, and refusing based on shard state rejects collections that
restored successfully, because activation in Overseer mode is asynchronous.
Details are on the PR. I’d welcome input on whether an explicit “restore in
progress” marker is the right direction.
> A failed RESTORE to a new collection leaves its shards in CONSTRUCTION, and
> retrying the restore doesn't fix them
> -----------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18584
> URL: https://issues.apache.org/jira/browse/SOLR-18584
> Project: Solr
> Issue Type: Bug
> Components: Backup/Restore
> Reporter: Boyi Dai
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When RESTORE targets a collection that doesn't exist, RestoreCmd
> (RestoreOnANewCollection) creates the collection, marks all its shards
> CONSTRUCTION, creates one replica per shard, installs the backed up data on
> each shard, and only then marks the shards ACTIVE. If installing the data
> fails for a shard, requestShardsToRestore() throws and
> markAllShardsAsActive() is never called. The collection is left in ZooKeeper
> with every shard in CONSTRUCTION.
> Unlike a failure to create the initial replicas, which deletes the
> collection, nothing cleans this up. Retrying the same RESTORE now finds the
> collection, so it takes the existing collection path
> (RestoreOnExistingCollection). That path restores the data and reports
> success, but never marks CONSTRUCTION shards ACTIVE. Updates to the
> collection are then rejected because no active shard covers their hash range.
> This was found through code analysis, and I reproduced it with a test using
> the test framework's error-throwing backup repository. It happens when
> RESTORE is called synchronously. An async RESTORE doesn't abort when
> restoring a shard fails (see the TODO in
> ShardRequestTracker.processResponses): it reports FAILED, but still marks the
> shards ACTIVE and leaves the collection in place without the data that failed
> to restore.
> *Steps to reproduce:*
> # Back up a collection.
> # Synchronously RESTORE the backup to a new collection T, with restoring the
> shard data failing (e.g. the backup repository fails to copy an index file).
> # Retry the same RESTORE once the repository works again.
> # Result: the retry succeeds, but T's shards stay in CONSTRUCTION, so updates
> to T fail.
> *Expected:* a failed restore to a new collection shouldn't leave a
> half-restored collection behind, and a retry should produce a usable
> collection.
> *Proposed fix:* if anything fails after the collection is created and before
> its shards are marked ACTIVE, delete the collection, as is already done when
> the initial replicas can't be created, and treat a shard restore failure
> recorded by an async request the same way. A retry then restores to a new
> collection again. I have a patch with a test and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
