Anon2Tokyo commented on issue #16375: URL: https://github.com/apache/dubbo/issues/16375#issuecomment-5282949752
@AcceptMediocrity hello bro. So far, the cause of the problem has been identified as follows: After a Pod in K8s is restarted, the IP address of the service instance changes. However, in some cases, the Dubbo consumer still retains the old Pod IP address, causing attempts to connect to an no longer existent address to continue. The specific trigger point lies in the service discovery notification mechanism: when Nacos/K8s sends a service discovery notification, ServiceInstancesChangedListener receives the list of new instances. The listener then retrieves the metadata based on the revision of each instance. If all versions of the metadata are failed to be retrieved during a notification, the old logic will directly return. This premature return prevents further notifications from being sent to downstream components. Although the downstream ServiceDiscoveryRegistryDirectory has the capability to clean up unused invokers using destroyUnusedInvokers(), it doesn’t receive any notifications indicating that the address has changed or been replaced. Therefore, the old IP addres s remains in use. **The solution** is to remove the premature return of ServiceInstancesChangedListener when “all metadata is empty/failed”. This means that errors are still recorded, retries are still attempted, but the process continues with the reconstruction of service URLs and notification of changes in the address. In this way, even if the metadata retrieval fails temporarily, the status of the currently resolvable address is still notified. For the old service addresses, subsequent directory refreshes will return either an empty or new address, thereby triggering the elimination of the old invoker, and preventing continued connection to the old Pod IP address. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
