Gallardot opened a new issue, #14825: URL: https://github.com/apache/dolphinscheduler/issues/14825
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened We found some exception logs on the worker server. ``` [INFO] 2023-08-28 14:55:23.165 +0800 org.apache.dolphinscheduler.server.worker.message.MessageRetryRunner:[129] - [WorkflowInstance-0][TaskInstance-954] - Begin retry send message to master, event: TaskInstanceExecutionRunningEvent(taskInstanceId=954, processInstanceId=1545, startTime=1693202790052, host=dolphinscheduler-worker-0.dolphinscheduler-worker-headless:1234, status=TaskExecutionStatus{code=1, desc='running'}, logPath=/opt/dolphinscheduler/logs/20230821/10645509557920/1/1545/954.log, executePath=null, processId=0, appIds=null, eventCreateTime=0, eventSendTime=1693205422750) [ERROR] 2023-08-28 14:55:23.166 +0800 org.apache.dolphinscheduler.extract.base.server.JdkDynamicServerHandler:[84] - [WorkflowInstance-0][TaskInstance-0] - Cannot find the ServerMethodInvoker of : Transporter(header=TransporterHeader(methodIdentifier=public abstract void org.apache.dolphinscheduler.extract.master.ITaskInstanceExecutionEventListener.onTaskInstanceExecutionRunning(org.apache.dolphinscheduler.extract.master.transportor.TaskInstanceExecutionRunningEvent), opaque=22), body=[123, 34, 97, 114, 103, 115, 34, 58, 91, 34, 101, 121, 74, 48, 89, 88, 78, 114, 83, 87, 53, 122, 100, 71, 70, 117, 89, 50, 86, 74, 90, 67, 73, 54, 79, 84, 85, 48, 76, 67, 74, 119, 99, 109, 57, 106, 90, 88, 78, 122, 83, 87, 53, 122, 100, 71, 70, 117, 89, 50, 86, 74, 90, 67, 73, 54, 77, 84, 85, 48, 78, 83, 119, 105, 99, 51, 82, 104, 99, 110, 82, 85, 97, 87, 49, 108, 73, 106, 111, 120, 78, 106, 107, 122, 77, 106, 65, 121, 78, 122, 107, 119, 77, 68, 85, 121, 76, 67, 74, 111, 98, 51, 78, 48, 73, 106, 111, 105, 90, 71, 57, 115, 99, 71, 104, 112, 98, 110, 78, 106, 97, 71, 86, 107, 100, 87, 120, 108, 99, 105, 49, 51, 98, 51, 74, 114, 90, 88, 73, 116, 77, 67, 53, 107, 98, 50, 120, 119, 97, 71, 108, 117, 99, 50, 78, 111, 90, 87, 82, 49, 98, 71, 86, 121, 76, 88, 100, 118, 99, 109, 116, 108, 99, 105, 49, 111, 90, 87, 70, 107, 98, 71, 86, 122, 99, 122, 111, 120, 77, 106, 77, 48, 73, 105, 119, 105, 99, 51, 82, 104, 100, 72, 86, 122, 73, 106, 111, 105, 85, 108, 86, 79, 84, 107, 108, 79, 82, 49, 57, 70, 87, 69, 86, 68, 86, 86, 82, 74, 84, 48, 52, 105, 76, 67, 74, 115, 98, 50, 100, 81, 89, 88, 82, 111, 73, 106, 111, 105, 76, 50, 57, 119, 100, 67, 57, 107, 98, 50, 120, 119, 97, 71, 108, 117, 99, 50, 78, 111, 90, 87, 82, 49, 98, 71, 86, 121, 76, 50, 120, 118, 90, 51, 77, 118, 77, 106, 65, 121, 77, 122, 65, 52, 77, 106, 69, 118, 77, 84, 65, 50, 78, 68, 85, 49, 77, 68, 107, 49, 78, 84, 99, 53, 77, 106, 65, 118, 77, 83, 56, 120, 78, 84, 81, 49, 76, 122, 107, 49, 78, 67, 53, 115, 98, 50, 99, 105, 76, 67, 74, 108, 101, 71, 86, 106, 100, 88, 82, 108, 85, 71, 70, 48, 97, 67, 73, 54, 98, 110, 86, 115, 98, 67, 119, 105, 99, 72, 74, 118, 89, 50, 86, 122, 99, 48, 108, 107, 73, 106, 111, 119, 76, 67, 74, 104, 99, 72, 66, 74, 90, 72, 77, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 86, 50, 90, 87, 53, 48, 81, 51, 74, 108, 89, 88, 82, 108, 86, 71, 108, 116, 90, 83, 73, 54, 77, 67, 119, 105, 90, 88, 90, 108, 98, 110, 82, 84, 90, 87, 53, 107, 86, 71, 108, 116, 90, 83, 73, 54, 77, 84, 89, 53, 77, 122, 73, 119, 78, 84, 99, 121, 77, 122, 69, 50, 78, 88, 48, 61, 34, 93, 44, 34, 97, 114, 103, 115, 84, 121, 112, 101, 115, 34, 58, 91, 34, 111, 114, 103, 46, 97, 112, 97, 99, 104, 101, 46, 100, 111, 108, 112, 104, 105, 110, 115, 99, 104, 101, 100, 117, 108, 101, 114, 46, 101, 120, 116, 114, 97, 99, 116, 46, 109, 97, 115, 116, 101, 114, 46, 116, 114, 97, 110, 115, 112, 111, 114, 116, 111, 114, 46, 84, 97, 115, 107, 73, 110, 115, 116, 97, 110, 99, 101, 69, 120, 101, 99, 117, 116, 105, 111, 110, 8 2, 117, 110, 110, 105, 110, 103, 69, 118, 101, 110, 116, 34, 93, 125]) [WARN] 2023-08-28 14:55:23.167 +0800 org.apache.dolphinscheduler.server.worker.message.MessageRetryRunner:[136] - [WorkflowInstance-0][TaskInstance-954] - Retry send message to master error org.apache.dolphinscheduler.extract.base.exception.MethodInvocationException: Cannot find the ServerMethodInvoker of public abstract void org.apache.dolphinscheduler.extract.master.ITaskInstanceExecutionEventListener.onTaskInstanceExecutionRunning(org.apache.dolphinscheduler.extract.master.transportor.TaskInstanceExecutionRunningEvent) at org.apache.dolphinscheduler.extract.base.exception.MethodInvocationException.of(MethodInvocationException.java:27) at org.apache.dolphinscheduler.extract.base.client.SyncClientMethodInvoker.invoke(SyncClientMethodInvoker.java:48) at org.apache.dolphinscheduler.extract.base.client.ClientInvocationHandler.invoke(ClientInvocationHandler.java:55) at com.sun.proxy.$Proxy167.onTaskInstanceExecutionRunning(Unknown Source) at org.apache.dolphinscheduler.server.worker.message.TaskInstanceExecutionRunningEventSender.sendEvent(TaskInstanceExecutionRunningEventSender.java:41) at org.apache.dolphinscheduler.server.worker.message.TaskInstanceExecutionRunningEventSender.sendEvent(TaskInstanceExecutionRunningEventSender.java:30) at org.apache.dolphinscheduler.server.worker.message.MessageRetryRunner.run(MessageRetryRunner.java:131) [INFO] 2023-08-28 14:55:24.168 +0800 org.apache.dolphinscheduler.server.worker.message.MessageRetryRunner:[129] - [WorkflowInstance-0][TaskInstance-954] - Begin retry send message to master, event: TaskInstanceExecutionFinishEvent(taskInstanceId=954, processInstanceId=1545, status=6, startTime=1693202790052, host=dolphinscheduler-worker-0.dolphinscheduler-worker-headless:1234, logPath=/opt/dolphinscheduler/logs/20230821/10645509557920/1/1545/954.log, executePath=/tmp/dolphinscheduler/exec/process/default/10579449480160/10645509557920_1/1545/954, endTime=1693202791555, processId=0, appIds=null, varPool=null, eventCreateTime=0, eventSendTime=1693205423753) [ERROR] 2023-08-28 14:55:24.169 +0800 org.apache.dolphinscheduler.extract.base.server.JdkDynamicServerHandler:[84] - [WorkflowInstance-0][TaskInstance-0] - Cannot find the ServerMethodInvoker of : Transporter(header=TransporterHeader(methodIdentifier=public abstract void org.apache.dolphinscheduler.extract.master.ITaskInstanceExecutionEventListener.onTaskInstanceExecutionFinish(org.apache.dolphinscheduler.extract.master.transportor.TaskInstanceExecutionFinishEvent), opaque=23), body=[123, 34, 97, 114, 103, 115, 34, 58, 91, 34, 101, 121, 74, 48, 89, 88, 78, 114, 83, 87, 53, 122, 100, 71, 70, 117, 89, 50, 86, 74, 90, 67, 73, 54, 79, 84, 85, 48, 76, 67, 74, 119, 99, 109, 57, 106, 90, 88, 78, 122, 83, 87, 53, 122, 100, 71, 70, 117, 89, 50, 86, 74, 90, 67, 73, 54, 77, 84, 85, 48, 78, 83, 119, 105, 99, 51, 82, 104, 100, 72, 86, 122, 73, 106, 111, 50, 76, 67, 74, 122, 100, 71, 70, 121, 100, 70, 82, 112, 98, 87, 85, 105, 79, 106, 69, 50, 79, 84, 77, 121, 77, 68, 73, 51, 79, 84, 65, 119, 7 8, 84, 73, 115, 73, 109, 104, 118, 99, 51, 81, 105, 79, 105, 74, 107, 98, 50, 120, 119, 97, 71, 108, 117, 99, 50, 78, 111, 90, 87, 82, 49, 98, 71, 86, 121, 76, 88, 100, 118, 99, 109, 116, 108, 99, 105, 48, 119, 76, 109, 82, 118, 98, 72, 66, 111, 97, 87, 53, 122, 89, 50, 104, 108, 90, 72, 86, 115, 90, 88, 73, 116, 100, 50, 57, 121, 97, 50, 86, 121, 76, 87, 104, 108, 89, 87, 82, 115, 90, 88, 78, 122, 79, 106, 69, 121, 77, 122, 81, 105, 76, 67, 74, 115, 98, 50, 100, 81, 89, 88, 82, 111, 73, 106, 111, 105, 76, 50, 57, 119, 100, 67, 57, 107, 98, 50, 120, 119, 97, 71, 108, 117, 99, 50, 78, 111, 90, 87, 82, 49, 98, 71, 86, 121, 76, 50, 120, 118, 90, 51, 77, 118, 77, 106, 65, 121, 77, 122, 65, 52, 77, 106, 69, 118, 77, 84, 65, 50, 78, 68, 85, 49, 77, 68, 107, 49, 78, 84, 99, 53, 77, 106, 65, 118, 77, 83, 56, 120, 78, 84, 81, 49, 76, 122, 107, 49, 78, 67, 53, 115, 98, 50, 99, 105, 76, 67, 74, 108, 101, 71, 86, 106, 100, 88, 82, 108, 85, 71, 70, 48, 97, 67, 73, 54, 73, 105, 57, 48, 98, 88, 65 , 118, 90, 71, 57, 115, 99, 71, 104, 112, 98, 110, 78, 106, 97, 71, 86, 107, 100, 87, 120, 108, 99, 105, 57, 108, 101, 71, 86, 106, 76, 51, 66, 121, 98, 50, 78, 108, 99, 51, 77, 118, 90, 71, 86, 109, 89, 88, 86, 115, 100, 67, 56, 120, 77, 68, 85, 51, 79, 84, 81, 48, 79, 84, 81, 52, 77, 68, 69, 50, 77, 67, 56, 120, 77, 68, 89, 48, 78, 84, 85, 119, 79, 84, 85, 49, 78, 122, 107, 121, 77, 70, 56, 120, 76, 122, 69, 49, 78, 68, 85, 118, 79, 84, 85, 48, 73, 105, 119, 105, 90, 87, 53, 107, 86, 71, 108, 116, 90, 83, 73, 54, 77, 84, 89, 53, 77, 122, 73, 119, 77, 106, 99, 53, 77, 84, 85, 49, 78, 83, 119, 105, 99, 72, 74, 118, 89, 50, 86, 122, 99, 48, 108, 107, 73, 106, 111, 119, 76, 67, 74, 104, 99, 72, 66, 74, 90, 72, 77, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 110, 90, 104, 99, 108, 66, 118, 98, 50, 119, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 86, 50, 90, 87, 53, 48, 81, 51, 74, 108, 89, 88, 82, 108, 86, 71, 108, 116, 90, 83, 73, 54, 77, 67, 119, 105, 90, 88, 90, 108, 98, 110, 8 2, 84, 90, 87, 53, 107, 86, 71, 108, 116, 90, 83, 73, 54, 77, 84, 89, 53, 77, 122, 73, 119, 78, 84, 99, 121, 78, 68, 69, 50, 79, 72, 48, 61, 34, 93, 44, 34, 97, 114, 103, 115, 84, 121, 112, 101, 115, 34, 58, 91, 34, 111, 114, 103, 46, 97, 112, 97, 99, 104, 101, 46, 100, 111, 108, 112, 104, 105, 110, 115, 99, 104, 101, 100, 117, 108, 101, 114, 46, 101, 120, 116, 114, 97, 99, 116, 46, 109, 97, 115, 116, 101, 114, 46, 116, 114, 97, 110, 115, 112, 111, 114, 116, 111, 114, 46, 84, 97, 115, 107, 73, 110, 115, 116, 97, 110, 99, 101, 69, 120, 101, 99, 117, 116, 105, 111, 110, 70, 105, 110, 105, 115, 104, 69, 118, 101, 110, 116, 34, 93, 125]) [WARN] 2023-08-28 14:55:24.171 +0800 org.apache.dolphinscheduler.server.worker.message.MessageRetryRunner:[136] - [WorkflowInstance-0][TaskInstance-954] - Retry send message to master error org.apache.dolphinscheduler.extract.base.exception.MethodInvocationException: Cannot find the ServerMethodInvoker of public abstract void org.apache.dolphinscheduler.extract.master.ITaskInstanceExecutionEventListener.onTaskInstanceExecutionFinish(org.apache.dolphinscheduler.extract.master.transportor.TaskInstanceExecutionFinishEvent) at org.apache.dolphinscheduler.extract.base.exception.MethodInvocationException.of(MethodInvocationException.java:27) at org.apache.dolphinscheduler.extract.base.client.SyncClientMethodInvoker.invoke(SyncClientMethodInvoker.java:48) at org.apache.dolphinscheduler.extract.base.client.ClientInvocationHandler.invoke(ClientInvocationHandler.java:55) at com.sun.proxy.$Proxy167.onTaskInstanceExecutionFinish(Unknown Source) at org.apache.dolphinscheduler.server.worker.message.TaskInstanceExecutionFinishEventSender.sendEvent(TaskInstanceExecutionFinishEventSender.java:39) at org.apache.dolphinscheduler.server.worker.message.TaskInstanceExecutionFinishEventSender.sendEvent(TaskInstanceExecutionFinishEventSender.java:28) at org.apache.dolphinscheduler.server.worker.message.MessageRetryRunner.run(MessageRetryRunner.java:131) ``` ### What you expected to happen no error ### How to reproduce A new cluster has been deployed through k8s with the dev branch, and MySQL is used as the database. server: 1 master, 1 worker. Both the master and worker were restarted at the same time, and then the worker produced the above log. @ruanwenjun @Radeity @EricGao888 PTAL. ### Anything else _No response_ ### Version dev ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
