Mr-j-yangyu opened a new issue, #18679:
URL: https://github.com/apache/dolphinscheduler/issues/18679

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### What happened
   
   <html>
   <body>
   <!--StartFragment--><!DOCTYPE html><p cid="n90" mdtype="paragraph" 
class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; 
orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; 
color: rgb(51, 51, 51); font-family: &quot;Open Sans&quot;, &quot;Clear 
Sans&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, &quot;Segoe UI 
Emoji&quot;, sans-serif; font-size: 16px; font-style: normal; 
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; 
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: 
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; 
text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;"><span md-inline="plain" class="md-plain 
md-expand" style="box-sizing: border-box;">In our production cluster (3.4.1), 
workflow instances can get permanently stuck in </span><span md-inline="code" 
spellcheck="false" class="md-pair-s" style="box-s
 izing: border-box;"><code style="box-sizing: border-box; font-family: 
var(--monospace); text-align: left; vertical-align: initial; border: 1px solid 
rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; 
padding: 0px 2px; font-size: 0.9em;">RUNNING_EXECUTION</code></span><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;"> even though 
</span><span md-inline="strong" class="md-pair-s " style="box-sizing: 
border-box;"><strong style="box-sizing: border-box;"><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;">all their task instances have 
already reached a final state</span></strong></span><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;"> (success/failure/killed) in 
the database.</span></p><p cid="n91" mdtype="paragraph" class="md-end-block 
md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 
0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); 
fon
 t-family: &quot;Open Sans&quot;, &quot;Clear Sans&quot;, &quot;Helvetica 
Neue&quot;, Helvetica, Arial, &quot;Segoe UI Emoji&quot;, sans-serif; 
font-size: 16px; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; 
text-align: start; text-indent: 0px; text-transform: none; widows: 2; 
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: 
initial; text-decoration-style: initial; text-decoration-color: initial;"><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;">This 
accumulated silently: every day a few daily-scheduled workflow instances 
finished all tasks but never transitioned to a final workflow state. After ~20 
days, the zombie instances reached </span><span md-inline="code" 
spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code 
style="box-sizing: border-box; font-family: var(--monospace); text-align: left; 
vertical-align: initial; border: 1px solid r
 gb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; 
padding: 0px 2px; font-size: 
0.9em;">master.server-load-protection.max-concurrent-workflow-instances</code></span><span
 md-inline="plain" class="md-plain" style="box-sizing: border-box;">, the 
master was marked overloaded (</span><span md-inline="code" spellcheck="false" 
class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: 
border-box; font-family: var(--monospace); text-align: left; vertical-align: 
initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 
244); border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">The current 
server is overload, cannot consumes commands.</code></span><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;">), and 
</span><span md-inline="strong" class="md-pair-s " style="box-sizing: 
border-box;"><strong style="box-sizing: border-box;"><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;">the who
 le cluster stopped consuming commands</span></strong></span><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;"> — hundreds 
of commands backlogged in </span><span md-inline="code" spellcheck="false" 
class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: 
border-box; font-family: var(--monospace); text-align: left; vertical-align: 
initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 
244); border-radius: 3px; padding: 0px 2px; font-size: 
0.9em;">t_ds_command</code></span><span md-inline="plain" class="md-plain" 
style="box-sizing: border-box;"> while all workers were idle.</span></p><p 
cid="n92" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: 
border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: 
pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: &quot;Open 
Sans&quot;, &quot;Clear Sans&quot;, &quot;Helvetica Neue&quot;, Helvetica, 
Arial, &quot;Segoe UI Emoji&quot;,
  sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: 
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; 
text-align: start; text-indent: 0px; text-transform: none; widows: 2; 
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: 
initial; text-decoration-style: initial; text-decoration-color: initial;"><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;">Example 
(workflow </span><span md-inline="code" spellcheck="false" class="md-pair-s" 
style="box-sizing: border-box;"><code style="box-sizing: border-box; 
font-family: var(--monospace); text-align: left; vertical-align: initial; 
border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); 
border-radius: 3px; padding: 0px 2px; font-size: 0.9em;">state = 
1</code></span><span md-inline="plain" class="md-plain" style="box-sizing: 
border-box;"> / RUNNING_EXECUTION, but 8/8 tasks already in final state, stuck 
for 20 days):</span></p><fig
 ure class="md-table-fig" cid="n93" mdtype="table" style="box-sizing: 
border-box; margin: 1.2em 0px; overflow-x: auto; max-width: calc(100% + 16px); 
padding: 0px; cursor: default; color: rgb(51, 51, 51); font-family: &quot;Open 
Sans&quot;, &quot;Clear Sans&quot;, &quot;Helvetica Neue&quot;, Helvetica, 
Arial, &quot;Segoe UI Emoji&quot;, sans-serif; font-size: 16px; font-style: 
normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 
400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; 
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; 
-webkit-text-stroke-width: 0px; text-decoration-thickness: initial; 
text-decoration-style: initial; text-decoration-color: initial;">
   workflow_instance_id | total tasks | finished tasks | unfinished tasks | 
stuck duration
   -- | -- | -- | -- | --
   96466 | 8 | 8 | 0 | 20 days
   106719 | 8 | 8 | 0 | 19 days
   
   </figure><p cid="n112" mdtype="paragraph" class="md-end-block md-p" 
style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 
0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); 
font-family: &quot;Open Sans&quot;, &quot;Clear Sans&quot;, &quot;Helvetica 
Neue&quot;, Helvetica, Arial, &quot;Segoe UI Emoji&quot;, sans-serif; 
font-size: 16px; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; 
text-align: start; text-indent: 0px; text-transform: none; widows: 2; 
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: 
initial; text-decoration-style: initial; text-decoration-color: initial;"><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;">Root cause 
analysis based on the source code:</span></p><p cid="n113" mdtype="paragraph" 
class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; 
orphans: 4; margin: 0.8em 0px
 ; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); 
font-family: &quot;Open Sans&quot;, &quot;Clear Sans&quot;, &quot;Helvetica 
Neue&quot;, Helvetica, Arial, &quot;Segoe UI Emoji&quot;, sans-serif; 
font-size: 16px; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; 
text-align: start; text-indent: 0px; text-transform: none; widows: 2; 
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: 
initial; text-decoration-style: initial; text-decoration-color: initial;"><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;">The 
lifecycle event flow is:</span></p><ol class="ol-list" start="" cid="n114" 
mdtype="list" style="box-sizing: border-box; margin: 0.8em 0px; padding-left: 
30px; position: relative; color: rgb(51, 51, 51); font-family: &quot;Open 
Sans&quot;, &quot;Clear Sans&quot;, &quot;Helvetica Neue&quot;, Helvetica, 
Arial, &quot;Segoe UI Emoji&quot;, sans-ser
 if; font-size: 16px; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 
2; text-align: start; text-indent: 0px; text-transform: none; white-space: 
normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; 
text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;"><li class="md-list-item" cid="n115" 
mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: 
relative;"><p cid="n116" mdtype="paragraph" class="md-end-block md-p" 
style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 
0px 0.5rem; white-space: pre-wrap; position: relative;"><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;">Worker reports a task 
lifecycle event (in-memory queue + ACK retry on the worker side — this part is 
reliable).</span></p></li><li class="md-list-item" cid="n117" 
mdtype="list_item" style="box-sizing: border-box; 
 margin: 0px; position: relative;"><p cid="n118" mdtype="paragraph" 
class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; 
orphans: 4; margin: 0px 0px 0.5rem; white-space: pre-wrap; position: 
relative;"><span md-inline="plain" class="md-plain" style="box-sizing: 
border-box;">Master persists the task state to DB, then publishes follow-up 
events (workflow topology transition / finish) to the </span><span 
md-inline="strong" class="md-pair-s " style="box-sizing: border-box;"><strong 
style="box-sizing: border-box;"><span md-inline="plain" class="md-plain" 
style="box-sizing: border-box;">in-memory</span></strong></span><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;"> 
</span><span md-inline="code" spellcheck="false" class="md-pair-s" 
style="box-sizing: border-box;"><code style="box-sizing: border-box; 
font-family: var(--monospace); text-align: left; vertical-align: initial; 
border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244,
  244); border-radius: 3px; padding: 0px 2px; font-size: 
0.9em;">WorkflowEventBus</code></span><span md-inline="plain" class="md-plain" 
style="box-sizing: border-box;">.</span></p></li><li class="md-list-item" 
cid="n119" mdtype="list_item" style="box-sizing: border-box; margin: 0px; 
position: relative;"><p cid="n120" mdtype="paragraph" class="md-end-block md-p" 
style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 
0px 0.5rem; white-space: pre-wrap; position: relative;"><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;">Workflow finalization 
(</span><span md-inline="code" spellcheck="false" class="md-pair-s" 
style="box-sizing: border-box;"><code style="box-sizing: border-box; 
font-family: var(--monospace); text-align: left; vertical-align: initial; 
border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); 
border-radius: 3px; padding: 0px 2px; font-size: 
0.9em;">emitWorkflowFinishedEventIfApplicable</code></span><span md-in
 line="plain" class="md-plain" style="box-sizing: border-box;"> → </span><span 
md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: 
border-box;"><code style="box-sizing: border-box; font-family: 
var(--monospace); text-align: left; vertical-align: initial; border: 1px solid 
rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; 
padding: 0px 2px; font-size: 
0.9em;">WorkflowSucceedLifecycleEvent</code></span><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;"> / </span><span 
md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: 
border-box;"><code style="box-sizing: border-box; font-family: 
var(--monospace); text-align: left; vertical-align: initial; border: 1px solid 
rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; 
padding: 0px 2px; font-size: 
0.9em;">WorkflowFailedLifecycleEvent</code></span><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;">) is drive
 n purely by these in-memory events.</span></p></li></ol><p cid="n121" 
mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; 
line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; 
position: relative; color: rgb(51, 51, 51); font-family: &quot;Open Sans&quot;, 
&quot;Clear Sans&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, 
&quot;Segoe UI Emoji&quot;, sans-serif; font-size: 16px; font-style: normal; 
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; 
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: 
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; 
text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;"><span md-inline="plain" class="md-plain" 
style="box-sizing: border-box;">If the master restarts (or event handling 
fails) between step 2 and step 3, the finish event is lost forever while the DB 
shows all tasks finished.</spa
 n></p><p cid="n122" mdtype="paragraph" class="md-end-block md-p" 
style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 
0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); 
font-family: &quot;Open Sans&quot;, &quot;Clear Sans&quot;, &quot;Helvetica 
Neue&quot;, Helvetica, Arial, &quot;Segoe UI Emoji&quot;, sans-serif; 
font-size: 16px; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; 
text-align: start; text-indent: 0px; text-transform: none; widows: 2; 
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: 
initial; text-decoration-style: initial; text-decoration-color: initial;"><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;">Failover 
does not heal this:</span></p><ul class="ul-list" cid="n123" mdtype="list" 
data-mark="-" style="box-sizing: border-box; margin: 0.8em 0px; padding-left: 
30px; position: relative; color: rgb(51,
  51, 51); font-family: &quot;Open Sans&quot;, &quot;Clear Sans&quot;, 
&quot;Helvetica Neue&quot;, Helvetica, Arial, &quot;Segoe UI Emoji&quot;, 
sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: 
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; 
orphans: 2; text-align: start; text-indent: 0px; text-transform: none; 
white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 
0px; text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;"><li class="md-list-item" cid="n124" 
mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: 
relative;"><p cid="n125" mdtype="paragraph" class="md-end-block md-p" 
style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 
0px 0.5rem; white-space: pre-wrap; position: relative;"><span md-inline="code" 
spellcheck="false" class="md-pair-s" style="box-sizing: border-box;"><code 
style="box-sizing: border-box; font-fa
 mily: var(--monospace); text-align: left; vertical-align: initial; border: 1px 
solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 
3px; padding: 0px 2px; font-size: 
0.9em;">WorkflowFailoverCommandHandler#assembleWorkflowExecutionGraph</code></span><span
 md-inline="plain" class="md-plain" style="box-sizing: border-box;"> rebuilds 
the </span><span md-inline="code" spellcheck="false" class="md-pair-s" 
style="box-sizing: border-box;"><code style="box-sizing: border-box; 
font-family: var(--monospace); text-align: left; vertical-align: initial; 
border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); 
border-radius: 3px; padding: 0px 2px; font-size: 
0.9em;">WorkflowExecutionGraph</code></span><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;"> </span><span 
md-inline="strong" class="md-pair-s " style="box-sizing: border-box;"><strong 
style="box-sizing: border-box;"><span md-inline="plain" class="md-plain" 
style="box-sizing:
  border-box;">from existing task instances in the 
DB</span></strong></span><span md-inline="plain" class="md-plain" 
style="box-sizing: border-box;"> (their final states are restored), but no 
finish/transition events are re-generated for those already-finished 
tasks.</span></p></li><li class="md-list-item" cid="n126" mdtype="list_item" 
style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n127" 
mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; 
line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space: 
pre-wrap; position: relative;"><span md-inline="plain" class="md-plain" 
style="box-sizing: border-box;">After recovery, only a </span><span 
md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: 
border-box;"><code style="box-sizing: border-box; font-family: 
var(--monospace); text-align: left; vertical-align: initial; border: 1px solid 
rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; p
 adding: 0px 2px; font-size: 
0.9em;">WorkflowStartLifecycleEvent</code></span><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;"> is published. In </span><span 
md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: 
border-box;"><code style="box-sizing: border-box; font-family: 
var(--monospace); text-align: left; vertical-align: initial; border: 1px solid 
rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; 
padding: 0px 2px; font-size: 
0.9em;">WorkflowRunningStateAction#onStartEvent</code></span><span 
md-inline="plain" class="md-plain" style="box-sizing: 
border-box;">:</span></p></li></ul><pre class="md-fences md-end-block 
ty-contain-cm modeLoaded" spellcheck="false" lang="java" cid="n128" 
mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: 
var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; 
text-align: left; white-space: normal; background-image: inherit; 
background-positi
 on: inherit; background-size: inherit; background-repeat: inherit; 
background-attachment: inherit; background-origin: inherit; background-clip: 
inherit; background-color: rgb(248, 248, 248); position: relative !important; 
border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; 
margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); 
font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; 
font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; 
text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 
0px; text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;"><span role="presentation" style="box-sizing: 
border-box; padding-right: 0.1px;"><span class="cm-keyword" style="box-sizing: 
border-box; color: rgb(119, 0, 136);">final</span> <span class="cm-variable" 
style="box-sizing: border-box; color: rgb(0, 0, 0);">List</span><span class="cm
 -operator" style="box-sizing: border-box; color: rgb(152, 26, 
26);">&lt;</span><span class="cm-variable" style="box-sizing: border-box; 
color: rgb(0, 0, 0);">ITaskExecutionRunnable</span><span class="cm-operator" 
style="box-sizing: border-box; color: rgb(152, 26, 26);">&gt;</span> <span 
class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 
0);">startNodes</span> <span class="cm-operator" style="box-sizing: border-box; 
color: rgb(152, 26, 26);">=</span> <span class="cm-variable" style="box-sizing: 
border-box; color: rgb(0, 0, 0);">workflowExecutionGraph</span>.<span 
class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 
0);">getStartNodes</span>();</span><br><span role="presentation" 
style="box-sizing: border-box; padding-right: 0.1px;"><span class="cm-keyword" 
style="box-sizing: border-box; color: rgb(119, 0, 136);">if</span> (<span 
class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 
0);">startNodes</span>.<span class="cm-variable" style="b
 ox-sizing: border-box; color: rgb(0, 0, 0);">isEmpty</span>()) 
{</span><br><span role="presentation" style="box-sizing: border-box; 
padding-right: 0.1px;"> &nbsp; &nbsp;<span class="cm-variable" 
style="box-sizing: border-box; color: rgb(0, 0, 
0);">emitWorkflowFinishedEventIfApplicable</span>(<span class="cm-variable" 
style="box-sizing: border-box; color: rgb(0, 0, 
0);">workflowExecutionRunnable</span>); &nbsp;<span class="cm-comment" 
style="box-sizing: border-box; color: rgb(170, 85, 0);">// only when there is 
no start node</span></span><br><span role="presentation" style="box-sizing: 
border-box; padding-right: 0.1px;"> &nbsp; &nbsp;<span class="cm-keyword" 
style="box-sizing: border-box; color: rgb(119, 0, 
136);">return</span>;</span><br><span role="presentation" style="box-sizing: 
border-box; padding-right: 0.1px;">}</span><br><span role="presentation" 
style="box-sizing: border-box; padding-right: 0.1px;"><span class="cm-variable" 
style="box-sizing: border-box; color: rgb(0, 0, 0);
 ">triggerTasks</span>(<span class="cm-variable" style="box-sizing: border-box; 
color: rgb(0, 0, 0);">workflowExecutionRunnable</span>, <span 
class="cm-variable" style="box-sizing: border-box; color: rgb(0, 0, 
0);">startNodes</span>);</span></pre><p cid="n129" mdtype="paragraph" 
class="md-end-block md-p md-focus" style="box-sizing: border-box; line-height: 
inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: 
relative; color: rgb(51, 51, 51); font-family: &quot;Open Sans&quot;, 
&quot;Clear Sans&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, 
&quot;Segoe UI Emoji&quot;, sans-serif; font-size: 16px; font-style: normal; 
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; 
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: 
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; 
text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;"><span md-inline="plain" cl
 ass="md-plain" style="box-sizing: border-box;">For a recovered workflow whose 
start nodes exist and whose tasks are all finished, </span><span 
md-inline="code" spellcheck="false" class="md-pair-s" style="box-sizing: 
border-box;"><code style="box-sizing: border-box; font-family: 
var(--monospace); text-align: left; vertical-align: initial; border: 1px solid 
rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; 
padding: 0px 2px; font-size: 0.9em;">triggerTasks</code></span><span 
md-inline="plain" class="md-plain" style="box-sizing: border-box;"> is invoked, 
no new task events will ever arrive, the workflow event bus stays empty, and 
the workflow never finalizes. The recovered zombie occupies an entry of 
</span><span md-inline="code" spellcheck="false" class="md-pair-s" 
style="box-sizing: border-box;"><code style="box-sizing: border-box; 
font-family: var(--monospace); text-align: left; vertical-align: initial; 
border: 1px solid rgb(231, 234, 237); background-colo
 r: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: 
0.9em;">IWorkflowRepository</code></span><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;"> forever, eventually 
triggering the </span><span md-inline="code" spellcheck="false" 
class="md-pair-s" style="box-sizing: border-box;"><code style="box-sizing: 
border-box; font-family: var(--monospace); text-align: left; vertical-align: 
initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 
244); border-radius: 3px; padding: 0px 2px; font-size: 
0.9em;">max-concurrent-workflow-instances</code></span><span md-inline="plain" 
class="md-plain" style="box-sizing: border-box;"> overload protection and 
blocking the whole cluster.</span></p><!--EndFragment-->
   </body>
   </html>
   
   ### What you expected to happen
   
   After master failover/restart, a workflow whose task chains are all in final 
states should be finalized (success/failure/stop) instead of remaining in 
`RUNNING_EXECUTION` forever.
   
   A possible fix direction: in `WorkflowRunningStateAction#onStartEvent` (or 
during recovery), check the finish condition before `triggerTasks`:
   
   ```java
   // pseudo code
   if (workflowExecutionGraph.isAllTaskExecutionRunnableChainSuccess()
           || workflowExecutionGraph.isExistFailureTaskExecutionRunnableChain()
           || workflowExecutionGraph.isExistKilledTaskExecutionRunnableChain()) 
{
       emitWorkflowFinishedEventIfApplicable(workflowExecutionRunnable);
       return;
   }
   ```
   
   More generally, a periodic reconciliation between in-memory active workflows 
and DB task states would make the event-driven state machine self-healing.
   
   ### How to reproduce
   
   Deterministic production reproduction: in our cluster, a daily scheduled 
workflow produced a new zombie instance **every single day** for over 20 days 
(each instance ends with all 8 tasks finished, workflow stuck in 
RUNNING_EXECUTION). So on our deployment this is 100% reproducible per day.
   
   Minimal manual reproduction:
   
   1. Start a workflow with a few fast tasks (e.g. SHELL tasks).
   2. Watch the DB: when the last task instance transitions to a final state 
(`t_ds_task_instance.state` = 7), **immediately kill -9 the master process** 
(before the workflow finish event is processed by the workflow event bus).
   3. Restart the master. Failover recovers the workflow instance: all task 
instances are final, but the workflow instance stays `RUNNING_EXECUTION` 
forever, and no further events are ever generated for it.
   4. Repeat until `max-concurrent-workflow-instances` is reached → master 
stops consuming commands (`The current server is overload, cannot consumes 
commands.`), workers stay idle.
   
   Note: step 2 is a race, but in production with daily schedules it happens 
deterministically over time (master restarts, event handling failures, etc.).
   
   ### Anything else
   
   - Happens every day for daily scheduled workflows in our production; each 
restart of the master does not heal existing zombies.
   - Related but different: #15574 (recover workflow instance causes 
inconsistent workflow/task state).
   - Side finding: 
`master.server-load-protection.max-workflow-instance-runtime` (added in #17932) 
is declared in `MasterServerLoadProtectionConfig` but has **no caller** in the 
codebase, so it cannot mitigate this problem at the workflow level. It would be 
helpful to either implement it or correct the documentation.
   
   ### Version
   
   3.4.1
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: 
[email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to