janhoy opened a new pull request, #876: URL: https://github.com/apache/solr-operator/pull/876
Fixes #707. ## The bug `Reconcile` ran a cluster operation and then threw its error away (`err = nil`, added without explanation in #689). The operation handlers do not set `retryLaterDuration` when they fail, so a failed operation scheduled no requeue and nothing else was guaranteed to trigger another reconcile for that SolrCloud. One failed `CLUSTERSTATUS` call during a managed scale down therefore stalled the scale down indefinitely — including after Solr became reachable again. Discarding the error also made the `err != nil` check that decides whether to queue the operation for a later retry permanently false, so the "hit an error during operation" reason never fired. ## The fix Keep the operation's error in `clusterOpError` instead of dropping it: requeue after 5s when an operation fails and the handler asked for no requeue of its own, and let the retry-queue check see the error again. The error is still not returned from `Reconcile`, so the status update and the operation's own retry/give-up machinery keep working as before. This covers every cluster operation, not just scale down — scale up and rolling update had the same stall. ## Tests Three new regression tests drive `Reconcile` against a fake client with an HTTP transport that fails every Solr call. Without the fix the first and third fail with `RequeueAfter: 0s`: - a failed scale down is requeued and keeps its lock - a scale down that keeps failing is eventually unlocked and queued for a later retry - a failing long-timeout operation (PVC expansion) is queued for retry after a minute instead of holding its lock for ten ## Validation On a Kind cluster, with a 4-node SolrCloud holding a 4-shard collection, scaled down to 3 while a `hostAliases` entry black-holed the operator's route to the common service: - **before:** one `Error retrieving cluster status` line, then nothing for 5 minutes — lock held, StatefulSet still at 4 - **after:** retries every 5s, op queued and re-taken each minute; healing the route (pointing the black-hole Service at the Solr pods, so the operator pod was never restarted) had the replica migrated off `example-solrcloud-3` and the scale down complete 3 seconds later `make test`, `make vet` and `make check-helm` pass. ## Not fixed here `handleManagedCloudRollingUpdate` returns `requestInProgress = true` when the cluster-state fetch fails, even though no async request is in flight, so the retry-queue escape hatch at `!requestInProgress` is unreachable for that path. A managed rolling update against a persistently unreachable Solr therefore retries every 5s while holding `UpdateLock` rather than being queued for a later retry. That looks like a separate decision to make (the field means "an async Solr command is running"), so this PR leaves it alone — happy to change it here if you'd rather. --- 🤖 AI assisted -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
