HTFX-Cloud commented on issue #10965:
URL: https://github.com/apache/dubbo/issues/10965#issuecomment-1321409085
> > 出现异常状态时，A节点的日志存在【Consumer is shutting down】，关联的#6294
>
> 这个的完整报错堆栈可以贴一下吗
从10点开始出现Warn级别日志
[Dubbo-thread-2] ~ WARN
org.apache.dubbo.remoting.exchange.support.DefaultFuture 177 received - [DUBBO]
The timeout response finally returned at 2022-10-19 10:01:09.345, response
status is 20, channel: /10.177.56.133:43182 -> /10.177.56.133:20882, please
check provider side for detailed result., dubbo version: 2.7.8, current host:
161.2.56.133
以及error的完整信息
10-19 10:27:32,632 [pool-7-thread-305] ~db4ed7dab49b42dcaa809c8f212a2e5e
DEBUG a.b.c.bd.consumer.PrepareForTaskExecutor 110 prepareExecutionParameter -
DataOps: key=projectId,value=null
10-19 10:27:43,391 [pool-7-thread-4442] ~416ce361b8e9466f894a46a6e34adace
ERROR a.b.c.bd.manager.TaskExecutionService 199 executeTask - 1次调用后非文件存储异常信息
org.apache.dubbo.rpc.RpcException: Failed to invoke the method executeTask in
the service a.b.c.bd.api.IEngineService. Tried 1 times of the providers
[10.177.56.133:20882] (1/4) from the registry 10.177.56.130:9908 on the
consumer 161.2.56.133 using the dubbo version 2.7.8. Last error is: Fail to
invoke remote method: executeTask, provider:
dubbo://10.177.56.133:20882/engineService?anyhost=true&application=dbd-app&category=providers&check=false&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=public&init=true&interface=a.b.c.bd.api.IEngineService&metadata-type=remote&methods=reRunTask,executeTask&path=engineService&payload=104857600&pid=2378&protocol=dubbo&qos.enable=false®ister.ip=161.2.56.133&release=2.7.8&remote.application=dbd-app&retries=0&revision=5.3-MASTER-20220608.095739-141&side=co
nsumer&sticky=false&timeout=30000×tamp=1655987866520&version=5.0, cause:
java.lang.IllegalStateException: Consumer is shutting down and this call is
going to be stopped without receiving any result, usually this is called by a
slow provider instance or bad service implementation.
at
org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:113)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:260)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:88)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:74)
~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.common.bytecode.proxy0.executeTask(proxy0.java)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.common.bytecode.proxy0$$FastClassBySpringCGLIB$$c197671e.invoke(<generated>)
~[dubbo-2.7.8.jar:2.7.8]
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.apache.dubbo.common.bytecode.proxy0$$EnhancerBySpringCGLIB$$75f0c32c.executeTask(<generated>)
~[dubbo-2.7.8.jar:2.7.8]
at
a.b.c.bd.manager.TaskExecutionService.executeTask(TaskExecutionService.java:135)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.manager.TaskExecutionService.executeTask(TaskExecutionService.java:78)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.logic.impl.TodoTaskOperationService.processTodoTask(TodoTaskOperationService.java:415)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.logic.impl.TodoTaskOperationService.processBatchTodoTasks(TodoTaskOperationService.java:1065)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.logic.impl.TodoTaskOperationService.processBatchTodoTasksAsyncWithRerunTodos(TodoTaskOperationService.java:961)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.logic.impl.TodoTaskOperationService$$FastClassBySpringCGLIB$$29b976c.invoke(<generated>)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
a.b.c.bd.logic.impl.TodoTaskOperationService$$EnhancerBySpringCGLIB$$1e7ef4d6.processBatchTodoTasksAsyncWithRerunTodos(<generated>)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doEnd(DispatcherNodeInstanceHandler.java:794)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:400)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doTrigger(DispatcherNodeInstanceHandler.java:749)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:396)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doReleaseRes(DispatcherNodeInstanceHandler.java:681)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:392)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doReceiveVar(DispatcherNodeInstanceHandler.java:666)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:388)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doBusiness(DispatcherNodeInstanceHandler.java:596)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:384)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doCompeteRes(DispatcherNodeInstanceHandler.java:478)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:380)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doBegin(DispatcherNodeInstanceHandler.java:446)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:376)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.run(DispatcherNodeInstanceHandler.java:192)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalStateException: Consumer is shutting down and this call is
going to be stopped without receiving any result, usually this is called by a
slow provider instance or bad service implementation.
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
~[?:1.8.0_181]
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
~[?:1.8.0_181]
at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:181)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.invoke(AsyncToSyncInvoker.java:61)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:89)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:51)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:69)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:82)
~[dubbo-2.7.8.jar:2.7.8]
... 57 more
Caused by: java.lang.IllegalStateException: Consumer is shutting down and
this call is going to be stopped without receiving any result, usually this is
called by a slow provider instance or bad service implementation.
at
org.apache.dubbo.common.threadpool.ThreadlessExecutor.shutdownNow(ThreadlessExecutor.java:164)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.remoting.exchange.support.DefaultFuture.closeChannel(DefaultFuture.java:147)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.close(HeaderExchangeChannel.java:152)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.close(HeaderExchangeChannel.java:177)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.close(HeaderExchangeClient.java:144)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient.close(LazyConnectExchangeClient.java:204)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.close(ReferenceCountExchangeClient.java:164)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.destroy(DubboInvoker.java:153)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.destroy(AsyncToSyncInvoker.java:96)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.destroy(ListenerInvokerWrapper.java:89)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.destroy(ProtocolFilterWrapper.java:131)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.InvokerWrapper.destroy(InvokerWrapper.java:61)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.integration.RegistryDirectory.destroyUnusedInvokers(RegistryDirectory.java:580)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.integration.RegistryDirectory.refreshInvoker(RegistryDirectory.java:336)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.integration.RegistryDirectory.refreshOverrideAndInvoker(RegistryDirectory.java:271)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.integration.RegistryDirectory.notify(RegistryDirectory.java:254)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.support.AbstractRegistry.notify(AbstractRegistry.java:426)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.support.FailbackRegistry.doNotify(FailbackRegistry.java:407)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.support.FailbackRegistry.notify(FailbackRegistry.java:398)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.nacos.NacosRegistry.notifySubscriber(NacosRegistry.java:529)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.nacos.NacosRegistry.lambda$subscribeEventListener$10(NacosRegistry.java:507)
~[dubbo-2.7.8.jar:2.7.8]
at
com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:132)
~[nacos-client-1.3.0.jar:?]
... 3 more
10-19 10:27:32,627 [pool-7-thread-1405] ~720d2d06a8744456b4d740d5cd1be846
INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 410 doBegin -
NODE[720d2d06a8744456b4d740d5cd1be846] 执行任务开始步骤：初始化任务变量！
10-19 10:27:32,627 [pool-7-thread-4967] ~357eeb0785e2443a8d0f3594c8307ad5
DEBUG a.b.c.bd.dao.impl.TaskOperationDao 1503 getAllStatus - getAllStatus:[ ]
10-19 10:27:32,617 [pool-7-thread-2957] ~4031a4934f6c47cab380c66e2e46f4df
INFO a.b.c.bd.logic.impl.TodoTaskOperationService 269 processTodoTask -
处理代办时，workflowKey=03a7a64a0ce14b9b95d264036d964d89,workflowCurrentKey=69A7FA817268477B9C0DBF3686396C4F,nodeKey=fee5cfb39e9748de9fa152f55bc12b18,taskRunId=d1555c00314c4a759d333316327cecac对应的任务启动时的项目信息为{projectId=null,
spaceId=0, WfbloodRerunInfo=}
10-19 10:27:32,600 [pool-7-thread-1700] ~c885fe8c32434010a0eb9f06892c1b6f
INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 675 doReleaseRes -
NODE[c885fe8c32434010a0eb9f06892c1b6f] 执行任务释放资源步骤：释放资源！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscr...@dubbo.apache.org
For additional commands, e-mail: notifications-h...@dubbo.apache.org
