njnu-seafish commented on issue #17817:
URL:
https://github.com/apache/dolphinscheduler/issues/17817#issuecomment-3686259543
Both task-level and workflow-level timeout alerts are failing with
NullPointerException — first observed this morning.
>
[WI-0][TI-0] - 2025-12-23 16:17:50.839 INFO
[MasterRpcServer-methodInvoker-10] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish
event: TaskRunningLifecycleEvent{task=s1, runtimeContext=null}
[WI-174][TI-0] - 2025-12-23 16:17:50.944 INFO
[ds-workflow-eventbus-worker-3]
o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task s1
TaskRunningLifecycleEvent{task=s1, runtimeContext=null} with state
RUNNING_EXECUTION
[WI-174][TI-0] - 2025-12-23 16:18:50.631 INFO
[ds-workflow-eventbus-worker-8]
o.a.d.s.m.e.t.l.h.TaskTimeoutLifecycleEventHandler:[75] - The task s1
TimeoutStrategy is WARNFAILED, try to publish a kill event and send a timeout
alert.
[WI-174][TI-0] - 2025-12-23 16:18:50.632 INFO
[ds-workflow-eventbus-worker-8] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish
event: TaskKillLifecycleEvent{task=s1, delayTime=0}
[WI-174][TI-0] - 2025-12-23 16:18:50.632 ERROR
[ds-workflow-eventbus-worker-8] o.a.d.s.m.e.WorkflowEventBusFireWorker:[88] -
Fire event failed for WorkflowExecuteRunnable: s1-20251223161749852
org.apache.dolphinscheduler.server.master.engine.exceptions.WorkflowEventFireException:
Failed to fire event: TaskRetryEvent{task=s1, timeout=60004}
at
org.apache.dolphinscheduler.server.master.engine.WorkflowEventBusFireWorker.doFireSingleWorkflowEventBus(WorkflowEventBusFireWorker.java:133)
at
org.apache.dolphinscheduler.server.master.engine.WorkflowEventBusFireWorker.fireAllRegisteredEvent(WorkflowEventBusFireWorker.java:86)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException: null
at
org.apache.dolphinscheduler.dao.AlertDao.sendTaskTimeoutAlert(AlertDao.java:250)
at
org.apache.dolphinscheduler.service.alert.WorkflowAlertManager.sendTaskTimeoutAlert(WorkflowAlertManager.java:265)
at
org.apache.dolphinscheduler.server.master.engine.task.lifecycle.handler.TaskTimeoutLifecycleEventHandler.doTaskTimeoutAlert(TaskTimeoutLifecycleEventHandler.java:94)
at
org.apache.dolphinscheduler.server.master.engine.task.lifecycle.handler.TaskTimeoutLifecycleEventHandler.handle(TaskTimeoutLifecycleEventHandler.java:79)
at
org.apache.dolphinscheduler.server.master.engine.task.lifecycle.handler.TaskTimeoutLifecycleEventHandler.handle(TaskTimeoutLifecycleEventHandler.java:38)
at
org.apache.dolphinscheduler.server.master.engine.task.lifecycle.handler.AbstractTaskLifecycleEventHandler.handle(AbstractTaskLifecycleEventHandler.java:46)
at
org.apache.dolphinscheduler.server.master.engine.task.lifecycle.handler.AbstractTaskLifecycleEventHandler.handle(AbstractTaskLifecycleEventHandler.java:32)
at
org.apache.dolphinscheduler.server.master.engine.WorkflowEventBusFireWorker.doFireSingleEvent(WorkflowEventBusFireWorker.java:144)
at
org.apache.dolphinscheduler.server.master.engine.WorkflowEventBusFireWorker.doFireSingleWorkflowEventBus(WorkflowEventBusFireWorker.java:122)
... 8 common frames omitted
[WI-174][TI-0] - 2025-12-23 16:18:50.754 INFO
[ds-workflow-eventbus-worker-8]
o.a.d.s.m.e.t.c.PhysicalTaskExecutorClientDelegator:[170] - Kill task s1 on
executor 192.168.4.3:1234 successfully
[WI-174][TI-0] - 2025-12-23 16:18:50.755 INFO
[ds-workflow-eventbus-worker-8]
o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task s1
TaskKillLifecycleEvent{task=s1, delayTime=0} with state RUNNING_EXECUTION
[WI-0][TI-0] - 2025-12-23 16:19:01.864 INFO
[MasterRpcServer-methodInvoker-12] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish
event: TaskFailedLifecycleEvent{task=s1, endTime=Tue Dec 23 16:19:01 GMT+08:00
2025}
[WI-174][TI-0] - 2025-12-23 16:19:01.880 INFO
[ds-workflow-eventbus-worker-12] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish
event:
WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent{task=s1taskState=FAILURE}
[WI-174][TI-0] - 2025-12-23 16:19:01.883 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task s1
TaskFailedLifecycleEvent{task=s1, endTime=Tue Dec 23 16:19:01 GMT+08:00 2025}
with state RUNNING_EXECUTION
[WI-174][TI-0] - 2025-12-23 16:19:01.883 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[47] - Begin fire
workflow s1-20251223161749852
LifecycleEvent[WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent{task=s1taskState=FAILURE}]
with state: RUNNING_EXECUTION
[WI-174][TI-0] - 2025-12-23 16:19:01.883 INFO
[ds-workflow-eventbus-worker-12] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish
event: WorkflowFailedLifecycleEvent{workflow=s1-20251223161749852}
[WI-174][TI-0] - 2025-12-23 16:19:01.883 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[52] - Fired workflow
s1-20251223161749852
LifecycleEvent[WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent{task=s1taskState=FAILURE}]
with state: RUNNING_EXECUTION
[WI-174][TI-0] - 2025-12-23 16:19:01.883 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[47] - Begin fire
workflow s1-20251223161749852
LifecycleEvent[WorkflowFailedLifecycleEvent{workflow=s1-20251223161749852}]
with state: RUNNING_EXECUTION
[WI-174][TI-0] - 2025-12-23 16:19:01.888 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.w.s.AbstractWorkflowStateAction:[161] - Success set
WorkflowExecuteRunnable: s1-20251223161749852 state from: RUNNING_EXECUTION to
FAILURE
[WI-174][TI-0] - 2025-12-23 16:19:01.888 INFO
[ds-workflow-eventbus-worker-12] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish
event: WorkflowFinalizeLifecycleEvent{workflow=s1-20251223161749852}
[WI-174][TI-0] - 2025-12-23 16:19:01.888 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[52] - Fired workflow
s1-20251223161749852
LifecycleEvent[WorkflowFailedLifecycleEvent{workflow=s1-20251223161749852}]
with state: FAILURE
[WI-174][TI-0] - 2025-12-23 16:19:01.888 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[47] - Begin fire
workflow s1-20251223161749852
LifecycleEvent[WorkflowFinalizeLifecycleEvent{workflow=s1-20251223161749852}]
with state: FAILURE
[WI-174][TI-0] - 2025-12-23 16:19:01.888 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.w.s.AbstractWorkflowStateAction:[204] -
********************************************************************************
Workflow Instance Detail
********************************************************************************
Workflow Instance Name: s1-20251223161749852
Command Type: START_PROCESS
State: FAILURE
StartNodes: [s1]
TotalTasks: 1
Host: 192.168.4.3:5678
Is SubWorkflow: NO
Run Times: 1
Tenant: default
Work Group: default
EventBusSummary: WorkflowEventBusSummary{eventCount=12,
fireSuccessEventCount=11, fireFailedEventCount=1}
Schedule Time: null
Start Time: Tue Dec 23 16:17:50 GMT+08:00 2025
Restart Time: Tue Dec 23 16:17:50 GMT+08:00 2025
End Time: Tue Dec 23 16:19:01 GMT+08:00 2025
[WI-174][TI-0] - 2025-12-23 16:19:01.895 INFO
[ds-workflow-eventbus-worker-12] o.a.d.d.AlertDao:[100] - add alert to db ,
alert: Alert(id=2, sign=6f782eb08d17c1b4e52e757f2171994be8bb1c11, title=start
workflow failed,
content=[{"projectCode":159047419031776,"projectName":"bug-dp","workflowInstanceId":174,"workflowDefinitionCode":160813687452064,"workflowInstanceName":"s1-20251223161749852","commandType":"START_PROCESS","workflowExecutionStatus":"FAILURE","modifyBy":"dolphinscheduler","recovery":"NO","runTimes":1,"workflowStartTime":"2025-12-23
16:17:50","workflowEndTime":"2025-12-23
16:19:01","workflowHost":"192.168.4.3:5678"}], alertStatus=null,
warningType=FAILURE, log=null, alertGroupId=2, createTime=Tue Dec 23 16:19:01
GMT+08:00 2025, updateTime=null, projectCode=159047419031776,
workflowDefinitionCode=160813687452064, workflowInstanceId=174,
alertType=WORKFLOW_INSTANCE_FAILURE, info={})
[WI-174][TI-0] - 2025-12-23 16:19:01.895 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.w.s.AbstractWorkflowStateAction:[210] - Successfully finalize
WorkflowExecuteRunnable: s1-20251223161749852
[WI-174][TI-0] - 2025-12-23 16:19:01.895 INFO
[ds-workflow-eventbus-worker-12]
o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[52] - Fired workflow
s1-20251223161749852
LifecycleEvent[WorkflowFinalizeLifecycleEvent{workflow=s1-20251223161749852}]
with state: FAILURE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
