HoustonPutman opened a new pull request, #910:
URL: https://github.com/apache/solr-operator/pull/910

   Fixes #824
   
   **This replaces #846.** #846 treats a `notfound` async status as a terminal 
failure. This PR instead submits the backup again, so a recurring backup keeps 
going and still produces a backup, rather than recording a failed run. It 
builds on #907, which handles a backup's async ID being present in Solr before 
it is submitted.
   
   ### The problem
   A collection backup whose async status Solr reports as `notfound` stayed 
`inProgress` forever. Because the SolrBackup never finished, a recurring backup 
was never scheduled again and `maxSaved` pruning stopped. Nothing was reported.
   
   ### Changes
   - **Resubmit on `notfound`.** Once a collection backup has been `notfound` 
for more than 5 seconds after it was submitted, it is submitted again, with the 
same name and async ID. Its start time is reset on every submission, so the 5 
second grace also applies after a resubmission.
   - **Delete the async status only after the result is saved.** Solr's async 
status for a finished collection backup used to be deleted before the 
SolrBackup status recording the result was saved. If that save failed, the next 
reconcile found the backup still in progress, with Solr reporting it as 
`notfound`. That is one way into #824.
     - `Reconcile` now fetches the SolrCloud and builds its auth context.
     - `reconcileSolrCloudBackup` returns the collections whose async requests 
finished.
     - `Reconcile` deletes their async statuses only once the status patch 
succeeds.
   - **Don't finish without a collection that failed to start.** If a 
collection's BACKUP call failed, its status was never recorded. Once the other 
collections finished, the SolrBackup was marked finished, and successful, 
without it. Its status is now recorded as not started, so the SolrBackup waits 
for it.
   - **Error handling.**
     - A failure for one collection no longer stops the other collections from 
being checked. The errors are returned together.
     - A failure to list the collections ends the reconcile.
     - Errors are only recorded as events once, as `BackupError` by 
`Reconcile`. The duplicate `BackupCloudNotReady` and 
`BackupCollectionListError` events are removed. Events were added in #836, 
which has not been released.
   
   ### Testing
   - New unit tests in `controllers/solrbackup_controller_unit_test.go`, 
against a fake Solr Collections API:
     - resubmitting a `notfound` backup and seeing it through to completion;
     - no resubmit within the grace period, including right after a 
resubmission;
     - the async status is deleted only after the status is saved, and kept 
when the save fails;
     - a collection that failed to start holds up the SolrBackup;
     - a failure to list collections ends the reconcile.
   - `events_test.go`: the cloud-not-ready test now goes through `Reconcile`, 
and checks that exactly one warning event is recorded.
   - #907's tests in `solrbackup_async_recovery_test.go` are updated for the 
deletion now happening after the status save, and for the status entry now 
being recorded when the start fails.
   - Each new test fails without its fix. All plain controller tests pass, and 
`go vet` is clean, at every commit.
   - Not yet run: the envtest suite (`make test`) and the backup e2e tests 
(`make e2e-tests TEST_FILES=backups_test.go`).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to