Boyi Dai created SOLR-18584:
-------------------------------
Summary: A failed RESTORE to a new collection leaves its shards in
CONSTRUCTION, and retrying the restore doesn't fix them
Key: SOLR-18584
URL: https://issues.apache.org/jira/browse/SOLR-18584
Project: Solr
Issue Type: Bug
Reporter: Boyi Dai
When RESTORE targets a collection that doesn't exist, RestoreCmd
(RestoreOnANewCollection) creates the collection, marks all its shards
CONSTRUCTION, creates one replica per shard, installs the backed up data on
each shard, and only then marks the shards ACTIVE. If installing the data fails
for a shard, requestShardsToRestore() throws and markAllShardsAsActive() is
never called. The collection is left in ZooKeeper with every shard in
CONSTRUCTION.
Unlike a failure to create the initial replicas, which deletes the collection,
nothing cleans this up. Retrying the same RESTORE now finds the collection, so
it takes the existing collection path (RestoreOnExistingCollection). That path
restores the data and reports success, but never marks CONSTRUCTION shards
ACTIVE. Updates to the collection are then rejected because no active shard
covers their hash range.
This was found through code analysis, and I reproduced it with a test using the
test framework's error-throwing backup repository. It happens when RESTORE is
called synchronously. An async RESTORE doesn't abort when restoring a shard
fails (see the TODO in ShardRequestTracker.processResponses): it reports
FAILED, but still marks the shards ACTIVE and leaves the collection in place
without the data that failed to restore.
*Steps to reproduce:*
# Back up a collection.
# Synchronously RESTORE the backup to a new collection T, with restoring the
shard data failing (e.g. the backup repository fails to copy an index file).
# Retry the same RESTORE once the repository works again.
# Result: the retry succeeds, but T's shards stay in CONSTRUCTION, so updates
to T fail.
*Expected:* a failed restore to a new collection shouldn't leave a
half-restored collection behind, and a retry should produce a usable collection.
*Proposed fix:* if anything fails after the collection is created and before
its shards are marked ACTIVE, delete the collection, as is already done when
the initial replicas can't be created, and treat a shard restore failure
recorded by an async request the same way. A retry then restores to a new
collection again. I have a patch with a test and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
