Mr-j-yangyu opened a new issue, #18679: URL: https://github.com/apache/dolphinscheduler/issues/18679
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened <html> <body> <!--StartFragment--><!DOCTYPE html><p cid="n90" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span md-inline="plain" class="md-plain md-expand" style="box-sizing: border-box;">In our production cluster (3.4.1), workflow instances can get permanently stuck in </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-s izing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">RUNNING_EXECUTION</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> even though </span><span md-inline="strong" class="md-pair-s " style="box-sizing: border-box;"><strong style="box-sizing: border-box;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">all their task instances have already reached a final state</span></strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> (success/failure/killed) in the database.</span></p><p cid="n91" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); fon t-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">This accumulated silently: every day a few daily-scheduled workflow instances finished all tasks but never transitioned to a final workflow state. After ~20 days, the zombie instances reached </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid r gb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">master.server-load-protection.max-concurrent-workflow-instances</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">, the master was marked overloaded (</span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">The current server is overload, cannot consumes commands.</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">), and </span><span md-inline="strong" class="md-pair-s " style="box-sizing: border-box;"><strong style="box-sizing: border-box;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">the who le cluster stopped consuming commands</span></strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> — hundreds of commands backlogged in </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">t_ds_command</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> while all workers were idle.</span></p><p cid="n92" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Example (workflow </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">state = 1</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> / RUNNING_EXECUTION, but 8/8 tasks already in final state, stuck for 20 days):</span></p><fig ure class="md-table-fig" cid="n93" mdtype="table" style="box-sizing: border-box; margin: 1.2em 0px; overflow-x: auto; max-width: calc(100% + 16px); padding: 0px; cursor: default; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> workflow_instance_id | total tasks | finished tasks | unfinished tasks | stuck duration -- | -- | -- | -- | -- 96466 | 8 | 8 | 0 | 20 days 106719 | 8 | 8 | 0 | 19 days </figure><p cid="n112" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Root cause analysis based on the source code:</span></p><p cid="n113" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px ; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">The lifecycle event flow is:</span></p><ol class="ol-list" start="" cid="n114" mdtype="list" style="box-sizing: border-box; margin: 0.8em 0px; padding-left: 30px; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-ser if; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li class="md-list-item" cid="n115" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n116" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space: pre-wrap; position: relative;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Worker reports a task lifecycle event (in-memory queue + ACK retry on the worker side — this part is reliable).</span></p></li><li class="md-list-item" cid="n117" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n118" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space: pre-wrap; position: relative;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Master persists the task state to DB, then publishes follow-up events (workflow topology transition / finish) to the </span><span md-inline="strong" class="md-pair-s " style="box-sizing: border-box;"><strong style="box-sizing: border-box;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">in-memory</span></strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">WorkflowEventBus</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">.</span></p></li><li class="md-list-item" cid="n119" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n120" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space: pre-wrap; position: relative;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Workflow finalization (</span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">emitWorkflowFinishedEventIfApplicable</code></span><span md-in line="plain" class="md-plain" style="box-sizing: border-box;"> → </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">WorkflowSucceedLifecycleEvent</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> / </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">WorkflowFailedLifecycleEvent</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">) is drive n purely by these in-memory events.</span></p></li></ol><p cid="n121" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">If the master restarts (or event handling fails) between step 2 and step 3, the finish event is lost forever while the DB shows all tasks finished.</spa n></p><p cid="n122" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Failover does not heal this:</span></p><ul class="ul-list" cid="n123" mdtype="list" data-mark="-" style="box-sizing: border-box; margin: 0.8em 0px; padding-left: 30px; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li class="md-list-item" cid="n124" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n125" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space: pre-wrap; position: relative;"><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-fa mily: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">WorkflowFailoverCommandHandler#assembleWorkflowExecutionGraph</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> rebuilds the </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">WorkflowExecutionGraph</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> </span><span md-inline="strong" class="md-pair-s " style="box-sizing: border-box;"><strong style="box-sizing: border-box;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">from existing task instances in the DB</span></strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> (their final states are restored), but no finish/transition events are re-generated for those already-finished tasks.</span></p></li><li class="md-list-item" cid="n126" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n127" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space: pre-wrap; position: relative;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">After recovery, only a </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; p adding: 0px 2px; font-size: 0.9em;">WorkflowStartLifecycleEvent</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> is published. In </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">WorkflowRunningStateAction#onStartEvent</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">:</span></p></li></ul><pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="java" cid="n128" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-positi on: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;"><span class="cm-keyword" style="box-sizing: border-box; color: rgb(119, 0, 136);">final</span> <span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">List</span><span class="cm -operator" style="box-sizing: border-box; color: rgb(152, 26, 26);"><</span><span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">ITaskExecutionRunnable</span><span class="cm-operator" style="box-sizing: border-box; color: rgb(152, 26, 26);">></span> <span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">startNodes</span> <span class="cm-operator" style="box-sizing: border-box; color: rgb(152, 26, 26);">=</span> <span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">workflowExecutionGraph</span>.<span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">getStartNodes</span>();</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;"><span class="cm-keyword" style="box-sizing: border-box; color: rgb(119, 0, 136);">if</span> (<span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">startNodes</span>.<span class="cm-variable" style="b ox-sizing: border-box; color: rgb(0, 0, 0);">isEmpty</span>()) {</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;"> <span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">emitWorkflowFinishedEventIfApplicable</span>(<span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">workflowExecutionRunnable</span>); <span class="cm-comment" style="box-sizing: border-box; color: rgb(170, 85, 0);">// only when there is no start node</span></span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;"> <span class="cm-keyword" style="box-sizing: border-box; color: rgb(119, 0, 136);">return</span>;</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">}</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;"><span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0); ">triggerTasks</span>(<span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">workflowExecutionRunnable</span>, <span class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 0);">startNodes</span>);</span></pre><p cid="n129" mdtype="paragraph" class="md-end-block md-p md-focus" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><span md-inline="plain" cl ass="md-plain" style="box-sizing: border-box;">For a recovered workflow whose start nodes exist and whose tasks are all finished, </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">triggerTasks</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> is invoked, no new task events will ever arrive, the workflow event bus stays empty, and the workflow never finalizes. The recovered zombie occupies an entry of </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-colo r: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">IWorkflowRepository</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> forever, eventually triggering the </span><span md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); text-align: left; vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">max-concurrent-workflow-instances</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> overload protection and blocking the whole cluster.</span></p><!--EndFragment--> </body> </html> ### What you expected to happen After master failover/restart, a workflow whose task chains are all in final states should be finalized (success/failure/stop) instead of remaining in `RUNNING_EXECUTION` forever. A possible fix direction: in `WorkflowRunningStateAction#onStartEvent` (or during recovery), check the finish condition before `triggerTasks`: ```java // pseudo code if (workflowExecutionGraph.isAllTaskExecutionRunnableChainSuccess() || workflowExecutionGraph.isExistFailureTaskExecutionRunnableChain() || workflowExecutionGraph.isExistKilledTaskExecutionRunnableChain()) { emitWorkflowFinishedEventIfApplicable(workflowExecutionRunnable); return; } ``` More generally, a periodic reconciliation between in-memory active workflows and DB task states would make the event-driven state machine self-healing. ### How to reproduce Deterministic production reproduction: in our cluster, a daily scheduled workflow produced a new zombie instance **every single day** for over 20 days (each instance ends with all 8 tasks finished, workflow stuck in RUNNING_EXECUTION). So on our deployment this is 100% reproducible per day. Minimal manual reproduction: 1. Start a workflow with a few fast tasks (e.g. SHELL tasks). 2. Watch the DB: when the last task instance transitions to a final state (`t_ds_task_instance.state` = 7), **immediately kill -9 the master process** (before the workflow finish event is processed by the workflow event bus). 3. Restart the master. Failover recovers the workflow instance: all task instances are final, but the workflow instance stays `RUNNING_EXECUTION` forever, and no further events are ever generated for it. 4. Repeat until `max-concurrent-workflow-instances` is reached → master stops consuming commands (`The current server is overload, cannot consumes commands.`), workers stay idle. Note: step 2 is a race, but in production with daily schedules it happens deterministically over time (master restarts, event handling failures, etc.). ### Anything else - Happens every day for daily scheduled workflows in our production; each restart of the master does not heal existing zombies. - Related but different: #15574 (recover workflow instance causes inconsistent workflow/task state). - Side finding: `master.server-load-protection.max-workflow-instance-runtime` (added in #17932) is declared in `MasterServerLoadProtectionConfig` but has **no caller** in the codebase, so it cannot mitigate this problem at the workflow level. It would be helpful to either implement it or correct the documentation. ### Version 3.4.1 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
