AsperforMias opened a new issue, #3848:
URL: https://github.com/apache/dubbo-go/issues/3848
### What happened?
With application-level service discovery, after a provider restarts on the
same IP and port, the consumer keeps returning `No provider available` and
never recovers.
Expected: the consumer calls the restarted provider once it registers again.
Actual: the restarted instance is skipped, and every call fails. In the
reproduction below it was still failing more than 3 minutes later.
Consumer log after the restart:
```text
[Registry][Directory] selector add service
url{tri://192.168.1.10:20100/greet.GreetService?...}
[Registry][Directory] skip rebuilding closing instance due to tombstone,
instance key:
tri://:@192.168.1.10:20100/?interface=greet.GreetService&group=gray&version=1.0.0×tamp=&meshClusterID=
... call failed: failed to invoke the method Greet. No provider available
for the service ...
```
Cause:
1. On graceful shutdown the consumer receives the closing notification and
`RemoveClosingInstance` stores a closing tombstone for the instance key.
2. When the provider registers again, `doCacheInvoker` only lets the
instance through if its URL `timestamp` differs from the tombstone's. With
application-level discovery the URL is rebuilt from the `MetadataInfo` cached
by revision. The revision does not change across a restart, so the re-added URL
carries the old `timestamp` (`1795275198` both before and after the restart in
this case) and the instance is treated as a stale snapshot.
3. The tombstone expires after 30s, but no further registry event arrives,
so the instance is never added back. When all instances of the application go
away, Nacos does not push an empty list (`UpdateCacheWhenEmpty` is false by
default), so there is no delete event to clear the tombstone either.
Related: #3615 / #3625 have the same symptom with a different trigger
(transient metadata failure).
### How can we reproduce it? (optional)
1. Start Nacos, two providers of one application (Triple, ports 20100 and
20101) and one consumer calling them in a loop. Calls succeed.
2. Send SIGTERM to both providers and start them again on the same ports a
few seconds later.
3. The consumer logs `skip rebuilding closing instance due to tombstone` for
both instances and then fails every call with `No provider available`.
### Environment
- dubbo-go: develop (ba671017)
- Go 1.26.1, macOS
- Nacos 2.5.2, application-level service discovery, Triple protocol, default
graceful shutdown settings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]