HappinerWinnie opened a new issue, #15351: URL: https://github.com/apache/dubbo/issues/15351
### Pre-check - [x] I am sure that all the content I provide is in English. ### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues. ### Apache Dubbo Component Java SDK (apache/dubbo) ### Dubbo Version jdk8，dubbo2.7.4.1，微服务集群部署，动态更新副本 ### Steps to reproduce this issue 日志： `[2025-05-06T08:04:57.476+08:00] [WARN] [ConsumeMessageThread_17-706] [com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler] [xxxx-yeying-xxxx-shadow-service-prodk8s-w7w5v] [xxxxxx] [xxxx-yeying-xxxx-shadow-service] [0a07457c1746489897462779148] [1.9] [] [TaskHandler.java,245,com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler,invokeExecutorAndUpdateTask] [任务失败] ## ' org.apache.dubbo.rpc.RpcException: No provider available in [invoker :interface com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade -> zookeeper://B注册中心/org.apache.dubbo.registry.RegistryService?application=xxxx-yeying-xxxx-shadow-service&check=false&dubbo=2.0.2&interface=com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade&lazy=false&logger=log4j2&methods=queryProductPickItem,querySingleTableProductPickItem,queryPickedQty,insertProductPickItem&organization=xxxx&owner=xxxx-yeying-xxxx-shadow-service&pid=48&qos.enable=false®ister.ip=10.7.69.124& release=2.7.4.1&side=consumer&sticky=false&timeout=3000×tamp=1746457332224,directory: org.apache.dubbo.registry.integration.RegistryDirectory@66d2885c, invoker :interface com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade -> zookeeper://B注册中心/org.apache.dubbo.registry.RegistryService?anyhost=true&application=xxxx-yeying-xxxx-shadow-service&bean.name=ServiceBean:com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade&check=false&cluster=failfast&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade&lazy=false&loadbalance=random&logger=log4j2&methods=queryProductPickItem,querySingleTableProductPickItem,queryPickedQty,insertProductPickItem&organization=xxxx&owner=xxxx-yeying-xxxx-shadow-service&payload=60000000&pid=48&qos.enable=false®ister.ip=xxxx&release=2.7.4.1&remote.application=xxxx-yeying-product-xxxx-service&retries=0&rivers.t ags=zxcgray&side=consumer&sticky=false&timeout=3000×tamp=1746457332224,directory: org.apache.dubbo.registry.integration.RegistryDirectory@42576db9] at org.apache.dubbo.rpc.cluster.support.RegistryAwareClusterInvoker.doInvoke(RegistryAwareClusterInvoker.java:59) at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:248) at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:78) at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:55) at org.apache.dubbo.common.bytecode.proxy6.queryProductPickItem(proxy6.java) at com.xxxx.yeying.xxxx.shadow.service.integration.dubbo.ServiceStationPickClient.lambda$queryProductPickByRequestId$3b8e180$1(ServiceStationPickClient.java:89) at com.xxxx.zxc.xxxx.construction.utils.RpcClient.call(RpcClient.java:106) at com.xxxx.zxc.xxxx.construction.utils.RpcClient.call(RpcClient.java:96) at com.xxxx.zxc.xxxx.construction.utils.RpcCli ent.callForList(RpcClient.java:86) at com.xxxx.yeying.xxxx.shadow.service.integration.dubbo.ServiceStationPickClient.queryProductPickByRequestId(ServiceStationPickClient.java:89) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor.queryPickOrder(StationSnowTaskExecutor.java:718) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor.execute(StationSnowTaskExecutor.java:242) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor.execute(StationSnowTaskExecutor.java:100) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor$$FastClassBySpringCGLIB$$9b1bf0fc.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.frame work.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor$$EnhancerBySpringCGLIB$$61c7774b.execute() at com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler.invokeExecutorAndUpdateTask(TaskHandler.java:215) at com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler.submit(TaskHandler.java:95) at com.xxxx.yeying.xxxx.shadow.service.datax. TaskHandler$$FastClassBySpringCGLIB$$b528b9d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercep t(CglibAopProxy.java:691) at com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler$$EnhancerBySpringCGLIB$$692c73ce.submit() at com.xxxx.yeying.xxxx.shadow.service.datax.TaskManager.submit(TaskManager.java:110) at com.xxxx.yeying.xxxx.shadow.service.mq.PickToFinishPickOrderConsumer.onMessage(PickToFinishPickOrderConsumer.java:53) at com.xxxx.yeying.xxxx.shadow.service.mq.PickToFinishPickOrderConsumer.onMessage(PickToFinishPickOrderConsumer.java:25) at com.xxxx.yeying.xxxx.shadow.service.mq.PickToFinishPickOrderConsumer$$FastClassBySpringCGLIB$$6cdf58d2.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop .interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) at com.xxxx.logistics.zxc.metric.MetricAdvice.handle(MetricAdvice.java:46) at sun.reflect.GeneratedMethodAccessor956.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(As pectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at com.xxxx.yeying.xxxx.shadow.service.mq.PickToFinishPickOrderConsumer$$EnhancerBySpringCGLIB$$6eb79409.onMessage() at org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer$DefaultMessageListenerConcurrently.consumeMessage(DefaultRocketMQListenerContainer.java:295) at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at ja va.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)` apollo： dubbo.registry.address=A注册中心 dubbo.provider.loadbalance=random dubbo.provider.retries=0 dubbo.registries.isDefault=true dubbo.registries.B.address=B注册中心 I have determined that there is no code or configuration that instructs the service to go to registry B to find a provider, and that in most cases it goes to registry A to find a provider, and only rarely to registry B. It is suspected that `dubbo.registries.isDefault=true` is interfering with some of the behaviors of dubbo to cause the bug that is causing dubbo to go to the wrong registry to find a provider.It is suspected that `dubbo.registries.isDefault=true` interferes with some behavior of dubbo and causes a bug, which causes dubbo to go to the wrong registry to find a provider. 来自AI的解读： > 你观察到的现象——“大多数时候用对了 ZK，只有极少数情况用错了”——使得问题更加有趣和复杂。这确实强烈暗示了问题并非简单的静态配置错误，而是与某些动态条件、内部状态或潜在的 Bug 相关。 让我们深入思考为什么 dubbo.registries.isDefault=true 这个（可能不规范的）配置项会导致这种间歇性的错误行为： 非线程安全的初始化或状态变更: Dubbo 在启动时或运行时会根据配置初始化 Registry 和 RegistryDirectory 等组件。如果 dubbo.registries.isDefault=true 的处理逻辑存在并发问题（非线程安全），或者在处理 Apollo 配置动态更新时存在瑕疵，可能导致在某些竞争条件下，RegistryDirectory 错误地关联了 wanggezhan 注册中心。 由于竞争条件发生的概率较低，这就解释了为什么大多数时候是正常的，只有偶尔会出错。错误日志中出现两个 RegistryDirectory 实例 (@54463380 和 @fabef2e) 可能就是这种内部状态混乱的体现。 缓存与刷新机制: Dubbo 内部有各种缓存（如注册中心地址列表、服务提供者列表等）。RegistryDirectory 会定期或在收到通知时刷新这些缓存。 如果 dubbo.registries.isDefault=true 干扰了注册中心的订阅逻辑或缓存刷新机制，可能导致在某个时间点，RegistryDirectory 的内部缓存包含了来自 B 的（无效）信息，并在那一刻的调用中被使用。当缓存下次正确刷新后，又恢复正常。 与 Apollo 配置动态推送的交互: Apollo 的配置是可以动态更新并推送给应用的。当 Dubbo 相关的配置发生变更（即使只是 dubbo.registries.isDefault=true 这一项存在），Dubbo 内部会触发相应的更新逻辑。 如果这个更新逻辑处理 dubbo.registries.isDefault=true 时存在 Bug，可能会短暂地导致内部注册中心状态不一致，使得某些线程在那个瞬间获取到了错误的注册中心引用。 集群容错或路由策略的副作用: 虽然我们没有看到明确的路由规则，但 Dubbo 的某些集群容错策略（如 RegistryAwareCluster）或负载均衡策略在处理多注册中心信息时，如果因为 dubbo.registries.isDefault=true 导致内部数据结构混乱，可能会在某些边界条件下（例如某个 ZK 连接短暂抖动、网络分区）触发 fallback 或探测逻辑，错误地尝试了 B。 RegistryDirectory 的生命周期管理: RegistryDirectory 实例通常与引用的服务接口绑定。它的创建、销毁、更新过程如果受到 dubbo.registries.isDefault=true 的干扰，可能导致在某些情况下（例如 Bean 的重新创建、作用域变化），获取到了一个状态错误的 RegistryDirectory 实例。 总结: dubbo.registries.isDefault=true 这个配置很可能是触发 Dubbo 内部某个潜在 Bug 或非预期行为的诱因。这个 Bug 可能与多注册中心处理、动态配置更新、并发控制或缓存刷新有关，导致在特定（可能与时间、并发、配置变更等相关的）条件下，消费者错误地尝试了非默认的 B 注册中心。 ### What you expected to happen When there is no indication that an interface goes to a particular registry to find a provider, the default registry should be used instead of the unintended registry ### Anything else Occasionally, only 100 or so times in a day for hundreds of thousands of calls, so please tell me why! ### Are you willing to submit a pull request to fix on your own? - [x] Yes I am willing to submit a pull request on my own! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
