goto456 opened a new issue, #16573: URL: https://github.com/apache/dubbo/issues/16573
### Pre-check - [x] I am sure that all the content I provide is in English. ### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues. ### Apache Dubbo Component Java SPI Extensions (apache/dubbo-spi-extensions) ### Dubbo Version - Reviewed published artifact: `org.apache.dubbo.extensions:dubbo-registry-consul:3.2.0` (JAR and sources). - The same control flow is present in the extension repository at commit `705910bd9bdd9e8f42c436c2a5d1927d5f7a2876`. - This is a source/artifact-based defect report. I have not yet run an independent upstream end-to-end reproduction, so no specific JDK, OS, or Consul server version is asserted as a tested runtime environment. ### Steps to reproduce this issue ### Summary A recoverable exception from a background Consul blocking query can terminate the subscription's `ConsulNotifier` task. There is no automatic continuation/resubmission for that failed task, so the consumer can retain a stale provider list after Consul becomes reachable again. This concerns recovery of the **registry watch**, not RPC invocation retries or provider TTL heartbeat recovery. ### Proposed fault-injection reproduction The following is a proposed integration fault-injection procedure, **not a claim that it has already been executed end-to-end**: 1. Start a provider and consumer using the Consul registry. Route the consumer's Consul HTTP requests through a local test proxy that initially forwards normally. 2. Wait until the initial subscription and provider-list notification succeed. Keep that subscription active. 3. For one subsequent background query to `/v1/health/service/<service>`, have the proxy return HTTP 503 instead of forwarding it. Let the resulting Consul-client exception escape the query call. 4. Immediately restore normal forwarding, then change the service's provider registrations. 5. Observe whether further queries and provider-list notifications occur for the existing subscription. A deterministic unit test can instead provide a Consul-client stub that returns an initial valid response, throws one recoverable runtime exception on the first watch query, and would return updated data on later calls. Submit the actual notifier to an executor and inspect the returned Future in the test. The current control flow predicts exceptional task completion rather than a later query. ### What you expected to happen After a transient query failure, the active subscription should resume querying with bounded retry/backoff (or equivalent task recovery), and subsequent provider additions/removals should reach the consumer without restarting it. The watch failure should be observable in logs or task-health metrics. ### Anything else ### Actual behavior indicated by the implementation The following is confirmed by source/artifact inspection; the integration scenario above remains to be independently run: - [`doSubscribe()`](https://github.com/apache/dubbo-spi-extensions/blob/705910bd9bdd9e8f42c436c2a5d1927d5f7a2876/dubbo-registry-extensions/dubbo-registry-consul/src/main/java/org/apache/dubbo/registry/consul/ConsulRegistry.java#L149-L166) submits the notifier but discards the returned Future. - [`ConsulNotifier.run()`](https://github.com/apache/dubbo-spi-extensions/blob/705910bd9bdd9e8f42c436c2a5d1927d5f7a2876/dubbo-registry-extensions/dubbo-registry-consul/src/main/java/org/apache/dubbo/registry/consul/ConsulRegistry.java#L340-L349) directly loops over the processing methods without an exception boundary. - [The processing methods](https://github.com/apache/dubbo-spi-extensions/blob/705910bd9bdd9e8f42c436c2a5d1927d5f7a2876/dubbo-registry-extensions/dubbo-registry-consul/src/main/java/org/apache/dubbo/registry/consul/ConsulRegistry.java#L351-L376) invoke Consul queries and notifications without handling an escaping exception. Therefore an exception reaching `run()` ends that task. The executor captures the failure in its Future, but this caller does not inspect it. The executor's worker thread may survive; that does **not** mean the failed Runnable will automatically run again. A later explicit re-subscription may submit work again, but no autonomous recovery is visible in this path. ### Possible recovery considerations - Handle recoverable failures inside the watch lifecycle, with bounded backoff and failure logging; honor shutdown/interruption. - If supervising/resubmitting tasks, avoid duplicate watchers for the same subscription. - A blanket catch-and-continue is not sufficient for notification failures: the current methods advance `consulIndex` before conversion/listener delivery. Recovery should retain failed updates or otherwise ensure they are not skipped when the same index is returned. Could maintainers confirm whether another branch/release already addresses this failure mode, or whether recovery should be added to this Consul registry implementation? ### Related but different reports I searched open and closed issues and PRs in both `apache/dubbo` and `apache/dubbo-spi-extensions` and found no report of this specific watcher-task termination failure. [dubbo#8902](https://github.com/apache/dubbo/issues/8902) concerns provider TTL re-registration, and [dubbo-spi-extensions#183](https://github.com/apache/dubbo-spi-extensions/issues/183) concerns duplicate notifications. ### Do you have a (mini) reproduction demo? - [ ] Yes, I have a minimal reproduction demo to help resolve this issue more effectively! ### Are you willing to submit a pull request to fix on your own? - [ ] Yes I am willing to submit a pull request on my own! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
