AsperforMias opened a new issue, #3848:
URL: https://github.com/apache/dubbo-go/issues/3848

   ### What happened?
   
   With application-level service discovery, after a provider restarts on the 
same IP and port, the consumer keeps returning `No provider available` and 
never recovers.
   
   Expected: the consumer calls the restarted provider once it registers again.
   Actual: the restarted instance is skipped, and every call fails. In the 
reproduction below it was still failing more than 3 minutes later.
   
   Consumer log after the restart:
   
   ```text
   [Registry][Directory] selector add service 
url{tri://192.168.1.10:20100/greet.GreetService?...}
   [Registry][Directory] skip rebuilding closing instance due to tombstone, 
instance key: 
tri://:@192.168.1.10:20100/?interface=greet.GreetService&group=gray&version=1.0.0&timestamp=&meshClusterID=
   ... call failed: failed to invoke the method Greet. No provider available 
for the service ...
   ```
   
   Cause:
   
   1. On graceful shutdown the consumer receives the closing notification and 
`RemoveClosingInstance` stores a closing tombstone for the instance key.
   2. When the provider registers again, `doCacheInvoker` only lets the 
instance through if its URL `timestamp` differs from the tombstone's. With 
application-level discovery the URL is rebuilt from the `MetadataInfo` cached 
by revision. The revision does not change across a restart, so the re-added URL 
carries the old `timestamp` (`1795275198` both before and after the restart in 
this case) and the instance is treated as a stale snapshot.
   3. The tombstone expires after 30s, but no further registry event arrives, 
so the instance is never added back. When all instances of the application go 
away, Nacos does not push an empty list (`UpdateCacheWhenEmpty` is false by 
default), so there is no delete event to clear the tombstone either.
   
   Related: #3615 / #3625 have the same symptom with a different trigger 
(transient metadata failure).
   
   ### How can we reproduce it? (optional)
   
   1. Start Nacos, two providers of one application (Triple, ports 20100 and 
20101) and one consumer calling them in a loop. Calls succeed.
   2. Send SIGTERM to both providers and start them again on the same ports a 
few seconds later.
   3. The consumer logs `skip rebuilding closing instance due to tombstone` for 
both instances and then fails every call with `No provider available`.
   
   ### Environment
   
   - dubbo-go: develop (ba671017)
   - Go 1.26.1, macOS
   - Nacos 2.5.2, application-level service discovery, Triple protocol, default 
graceful shutdown settings
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to