HTFX-Cloud commented on issue #10965:
URL: https://github.com/apache/dubbo/issues/10965#issuecomment-1321403386
从10点开始出现[Dubbo-thread-2] ~ WARN
org.apache.dubbo.remoting.exchange.support.DefaultFuture 177 received -
[DUBBO] The timeout response finally returned at 2022-10-19 10:01:09.345,
response status is 20, channel: /10.177.56.133:43182 -> /10.177.56.133:20882,
please check provider side for detailed result., dubbo version: 2.7.8, current
host: 161.2.56.133
以及error的日志（该日志倒排取的，需要注意）
10-19 10:27:32,600 [pool-7-thread-1700] ~c885fe8c32434010a0eb9f06892c1b6f
INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 675 doReleaseRes -
NODE[c885fe8c32434010a0eb9f06892c1b6f] 执行任务释放资源步骤：释放资源！
10-19 10:27:32,617 [pool-7-thread-2957] ~4031a4934f6c47cab380c66e2e46f4df
INFO a.b.c.bd.logic.impl.TodoTaskOperationService 269 processTodoTask -
处理代办时，workflowKey=03a7a64a0ce14b9b95d264036d964d89,workflowCurrentKey=69A7FA817268477B9C0DBF3686396C4F,nodeKey=fee5cfb39e9748de9fa152f55bc12b18,taskRunId=d1555c00314c4a759d333316327cecac对应的任务启动时的项目信息为{projectId=null,
spaceId=0, WfbloodRerunInfo=}
10-19 10:27:32,627 [pool-7-thread-4967] ~357eeb0785e2443a8d0f3594c8307ad5
DEBUG a.b.c.bd.dao.impl.TaskOperationDao 1503 getAllStatus - getAllStatus:[ ]
10-19 10:27:32,627 [pool-7-thread-1405] ~720d2d06a8744456b4d740d5cd1be846
INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 410 doBegin -
NODE[720d2d06a8744456b4d740d5cd1be846] 执行任务开始步骤：初始化任务变量！
... 3 more
at
com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:132)
~[nacos-client-1.3.0.jar:?]
at
org.apache.dubbo.registry.nacos.NacosRegistry.lambda$subscribeEventListener$10(NacosRegistry.java:507)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.nacos.NacosRegistry.notifySubscriber(NacosRegistry.java:529)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.support.FailbackRegistry.notify(FailbackRegistry.java:398)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.support.FailbackRegistry.doNotify(FailbackRegistry.java:407)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.support.AbstractRegistry.notify(AbstractRegistry.java:426)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.integration.RegistryDirectory.notify(RegistryDirectory.java:254)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.integration.RegistryDirectory.refreshOverrideAndInvoker(RegistryDirectory.java:271)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.integration.RegistryDirectory.refreshInvoker(RegistryDirectory.java:336)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.registry.integration.RegistryDirectory.destroyUnusedInvokers(RegistryDirectory.java:580)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.InvokerWrapper.destroy(InvokerWrapper.java:61)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.destroy(ProtocolFilterWrapper.java:131)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.destroy(ListenerInvokerWrapper.java:89)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.destroy(AsyncToSyncInvoker.java:96)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.destroy(DubboInvoker.java:153)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.close(ReferenceCountExchangeClient.java:164)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient.close(LazyConnectExchangeClient.java:204)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.close(HeaderExchangeClient.java:144)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.close(HeaderExchangeChannel.java:177)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.close(HeaderExchangeChannel.java:152)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.remoting.exchange.support.DefaultFuture.closeChannel(DefaultFuture.java:147)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.common.threadpool.ThreadlessExecutor.shutdownNow(ThreadlessExecutor.java:164)
~[dubbo-2.7.8.jar:2.7.8]
Caused by: java.lang.IllegalStateException: Consumer is shutting down and
this call is going to be stopped without receiving any result, usually this is
called by a slow provider instance or bad service implementation.
... 57 more
at
org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:82)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:69)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:51)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:89)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.invoke(AsyncToSyncInvoker.java:61)
~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:181)
~[dubbo-2.7.8.jar:2.7.8]
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
~[?:1.8.0_181]
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
~[?:1.8.0_181]
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalStateException: Consumer is shutting down and this call is
going to be stopped without receiving any result, usually this is called by a
slow provider instance or bad service implementation.
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_181]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.run(DispatcherNodeInstanceHandler.java:192)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:376)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doBegin(DispatcherNodeInstanceHandler.java:446)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:380)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doCompeteRes(DispatcherNodeInstanceHandler.java:478)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:384)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doBusiness(DispatcherNodeInstanceHandler.java:596)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:388)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doReceiveVar(DispatcherNodeInstanceHandler.java:666)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:392)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doReleaseRes(DispatcherNodeInstanceHandler.java:681)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:396)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doTrigger(DispatcherNodeInstanceHandler.java:749)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:400)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.handler.DispatcherNodeInstanceHandler.doEnd(DispatcherNodeInstanceHandler.java:794)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.logic.impl.TodoTaskOperationService$$EnhancerBySpringCGLIB$$1e7ef4d6.processBatchTodoTasksAsyncWithRerunTodos(<generated>)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
a.b.c.bd.logic.impl.TodoTaskOperationService$$FastClassBySpringCGLIB$$29b976c.invoke(<generated>)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.logic.impl.TodoTaskOperationService.processBatchTodoTasksAsyncWithRerunTodos(TodoTaskOperationService.java:961)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.logic.impl.TodoTaskOperationService.processBatchTodoTasks(TodoTaskOperationService.java:1065)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.logic.impl.TodoTaskOperationService.processTodoTask(TodoTaskOperationService.java:415)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.manager.TaskExecutionService.executeTask(TaskExecutionService.java:78)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
a.b.c.bd.manager.TaskExecutionService.executeTask(TaskExecutionService.java:135)
~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at
org.apache.dubbo.common.bytecode.proxy0$$EnhancerBySpringCGLIB$$75f0c32c.executeTask(<generated>)
~[dubbo-2.7.8.jar:2.7.8]
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at
org.apache.dubbo.common.bytecode.proxy0$$FastClassBySpringCGLIB$$c197671e.invoke(<generated>)
~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.common.bytecode.proxy0.executeTask(proxy0.java)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:74)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:88)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:260)
~[dubbo-2.7.8.jar:2.7.8]
at
org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:113)
~[dubbo-2.7.8.jar:2.7.8]
10-19 10:27:43,391 [pool-7-thread-4442] ~416ce361b8e9466f894a46a6e34adace
ERROR a.b.c.bd.manager.TaskExecutionService 199 executeTask - 1次调用后非文件存储异常信息
org.apache.dubbo.rpc.RpcException: Failed to invoke the method executeTask in
the service a.b.c.bd.api.IEngineService. Tried 1 times of the providers
[10.177.56.133:20882] (1/4) from the registry 10.177.56.130:9908 on the
consumer 161.2.56.133 using the dubbo version 2.7.8. Last error is: Fail to
invoke remote method: executeTask, provider:
dubbo://10.177.56.133:20882/engineService?anyhost=true&application=dbd-app&category=providers&check=false&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=public&init=true&interface=a.b.c.bd.api.IEngineService&metadata-type=remote&methods=reRunTask,executeTask&path=engineService&payload=104857600&pid=2378&protocol=dubbo&qos.enable=false®ister.ip=161.2.56.133&release=2.7.8&remote.application=dbd-app&retries=0&revision=5.3-MASTER-20220608.095739-141&side=co
nsumer&sticky=false&timeout=30000×tamp=1655987866520&version=5.0, cause:
java.lang.IllegalStateException: Consumer is shutting down and this call is
going to be stopped without receiving any result, usually this is called by a
slow provider instance or bad service implementation.
10-19 10:27:32,632 [pool-7-thread-305] ~db4ed7dab49b42dcaa809c8f212a2e5e
DEBUG a.b.c.bd.consumer.PrepareForTaskExecutor 110 prepareExecutionParameter -
DataOps: key=projectId,value=null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscr...@dubbo.apache.org
For additional commands, e-mail: notifications-h...@dubbo.apache.org
