This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch 
gh-readonly-queue/dev/pr-13154-50cf6b6711f9316557daa3a5e17defdab4247953
in repository https://gitbox.apache.org/repos/asf/seatunnel.git

commit 2010a1fedbb20590ab18f69a52374478fc1e797d
Author: SEZ <[email protected]>
AuthorDate: Tue Nov 17 15:19:45 2026 +0000

    [Improve][Zeta] Retain why each pipeline restore happened (#13154)
    
    Co-authored-by: SEZ9 <[email protected]>
    Co-authored-by: Claude Opus 5 <[email protected]>
---
 docs/en/engines/zeta/rest-api-v2.md                |  10 ++
 docs/zh/engines/zeta/rest-api-v2.md                |   7 +
 .../engine/server/dag/physical/SubPlan.java        |  54 ++++++
 .../server/diagnostic/JobRuntimeDiagnostics.java   |  14 ++
 .../engine/server/dag/physical/SubPlanTest.java    | 195 +++++++++++++++++++++
 .../diagnostic/JobRuntimeDiagnosticsTest.java      |  53 ++++++
 6 files changed, 333 insertions(+)

diff --git a/docs/en/engines/zeta/rest-api-v2.md 
b/docs/en/engines/zeta/rest-api-v2.md
index c05f7ef575..91aef8cc5c 100644
--- a/docs/en/engines/zeta/rest-api-v2.md
+++ b/docs/en/engines/zeta/rest-api-v2.md
@@ -662,6 +662,10 @@ This endpoint helps troubleshoot why jobs stay in 
`PENDING` by showing the pendi
         "pipelineStatus": "RUNNING",
         "restoreCount": 7,
         "maxRestoreCount": 100,
+        "restoreCauses": [
+          "restore 6: 
org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: 
ErrorCode:[JDBC-05] ... Connection refused",
+          "restore 7: 
org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: 
ErrorCode:[JDBC-05] ... Connection refused"
+        ],
         "stateTimestamps": {
           "INITIALIZING": 1755000000000,
           "CREATED": 1755000000200,
@@ -684,6 +688,11 @@ node. It is auxiliary information: if it can not be 
obtained, the field is omitt
 failing the request. Only this endpoint returns it; `/running-jobs` does not, 
because collecting it
 for every running job would cost one more round trip to the master per job.
 
+`diagnostics` is read from the master's in-memory job state, so like 
`restoreCount` it describes
+this master's view of the job rather than its whole history. After a master 
switch the restore
+counters and `restoreCauses` start again from the recovered job, and causes 
recorded by the
+previous master are not carried over.
+
 #### Metrics field description
 
 | Field | Description |
@@ -715,6 +724,7 @@ for every running job would cost one more round trip to the 
master per job.
 | pipelines[].pipelineStatus | Current pipeline state |
 | pipelines[].restoreCount | How many times this pipeline has been restored 
since the job was submitted. A value that keeps growing while `jobStatus` stays 
`RUNNING` is a crash loop |
 | pipelines[].maxRestoreCount | Restore limit of this pipeline, from the 
`job.retry.times` env option |
+| pipelines[].restoreCauses | Why earlier runs of this pipeline ended, oldest 
first, one entry per restore. The engine keeps only the current attempt's 
error, so without this a crash loop can be detected from `restoreCount` but not 
explained. Each entry is truncated, and the list is capped at the last 16 
restores. A restore with no recorded error is still listed, because "restarted 
for an unknown reason" is a different state from "restarted because the sink 
refused the connection" |
 | pipelines[].stateTimestamps | Epoch millis when the pipeline entered each 
state. After a restore, the timestamps of the new attempt overwrite the 
previous ones |
 | totalPipelineRestoreCount | Sum of `restoreCount` over all pipelines of the 
job |
 
diff --git a/docs/zh/engines/zeta/rest-api-v2.md 
b/docs/zh/engines/zeta/rest-api-v2.md
index 910b90d78f..b233c47a12 100644
--- a/docs/zh/engines/zeta/rest-api-v2.md
+++ b/docs/zh/engines/zeta/rest-api-v2.md
@@ -653,6 +653,10 @@ seatunnel:
         "pipelineStatus": "RUNNING",
         "restoreCount": 7,
         "maxRestoreCount": 100,
+        "restoreCauses": [
+          "restore 6: 
org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: 
ErrorCode:[JDBC-05] ... Connection refused",
+          "restore 7: 
org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: 
ErrorCode:[JDBC-05] ... Connection refused"
+        ],
         "stateTimestamps": {
           "INITIALIZING": 1755000000000,
           "CREATED": 1755000000200,
@@ -672,6 +676,8 @@ seatunnel:
 `finishTime`, `errorMsg` 字段在Job结束时会返回,结束状态为不为RUNNING,可能为FINISHED,可能为CANCEL
 `diagnostics` 
字段在Job运行中且能从Master节点读取到诊断信息时返回。它属于辅助信息,读取失败时该字段会被省略,不会导致请求失败。该字段只在本接口返回,`/running-jobs`
 不返回:为列表中的每个Job收集诊断信息会额外增加一次到Master的请求。
 
+`diagnostics` 读取的是 Master 内存中的 Job 状态,因此与 `restoreCount` 一样,它描述的是当前这个 Master 
对该 Job 的视图,而不是完整历史。Master 切换后,恢复计数与 `restoreCauses` 会从被恢复的 Job 重新开始,上一个 Master 
记录的原因不会延续。
+
 #### 指标字段说明
 
 | 字段 | 说明 |
@@ -703,6 +709,7 @@ seatunnel:
 | pipelines[].pipelineStatus | Pipeline 当前状态 |
 | pipelines[].restoreCount | 自 Job 提交以来该 Pipeline 被恢复(重启)的次数。若 `jobStatus` 一直是 
`RUNNING` 而该值不断增长,说明处于崩溃重启循环中 |
 | pipelines[].maxRestoreCount | 该 Pipeline 的恢复次数上限,来自 `job.retry.times` env 配置 
|
+| pipelines[].restoreCauses | 该 Pipeline 
此前每一次运行结束的原因,按时间从旧到新,每次恢复一条。引擎只保留当前这次尝试的错误,因此若没有该字段,仅凭 `restoreCount` 
只能发现崩溃重启循环、无法知道原因。每条都会截断,且只保留最近 16 次恢复。未记录到错误的那次恢复也会列出,因为「因未知原因重启」与「因 Sink 
拒绝连接而重启」是两种不同的状态 |
 | pipelines[].stateTimestamps | Pipeline 进入各状态的时间戳(毫秒)。发生恢复后,新一轮的时间戳会覆盖上一轮 |
 | totalPipelineRestoreCount | Job 下所有 Pipeline 的 `restoreCount` 之和 |
 
diff --git 
a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java
 
b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java
index c4ee893ad4..008d2314da 100644
--- 
a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java
+++ 
b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java
@@ -43,8 +43,11 @@ import lombok.Data;
 import lombok.NonNull;
 import lombok.extern.slf4j.Slf4j;
 
+import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicReference;
@@ -94,6 +97,27 @@ public class SubPlan {
     /** The error throw by physicalVertex, should be set when physicalVertex 
throw error. */
     private AtomicReference<String> errorByPhysicalVertex = new 
AtomicReference<>();
 
+    /**
+     * Why each earlier run of this pipeline ended, oldest first.
+     *
+     * <p>{@link #errorByPhysicalVertex} only holds the current attempt and is 
replaced on every
+     * restore, so by the time anyone looks at a pipeline that has restarted 
repeatedly the reasons
+     * are gone. The restore counter then says a crash loop is happening 
without saying what is
+     * causing it, which is the first thing an operator needs.
+     *
+     * <p>The size is bounded twice over: a restore only happens while {@code 
pipelineRestoreNum <
+     * pipelineMaxRestoreNum}, which {@code job.retry.times} sets and defaults 
to 3, and {@link
+     * #MAX_RETAINED_RESTORE_CAUSES} caps it regardless in case that is 
configured high. Each entry
+     * is truncated, so the whole list stays small enough to live beside the 
pipeline.
+     */
+    private final Queue<String> restoreCauses = new ConcurrentLinkedQueue<>();
+
+    /** Upper bound on {@link #restoreCauses}, independent of {@code 
job.retry.times}. */
+    private static final int MAX_RETAINED_RESTORE_CAUSES = 16;
+
+    /** Longest retained cause. Enough for a message and the first frames that 
matter. */
+    private static final int MAX_RESTORE_CAUSE_LENGTH = 2048;
+
     private final ExecutorService executorService;
 
     private JobMaster jobMaster;
@@ -550,6 +574,33 @@ public class SubPlan {
                         Constant.OPERATION_RETRY_SLEEP));
     }
 
+    /**
+     * Keep why a run ended, dropping the oldest when full.
+     *
+     * <p>A restore with no recorded error is still worth a placeholder: 
"restarted and we do not
+     * know why" is a different and more alarming state than "restarted 
because the sink refused the
+     * connection", and silently omitting it would make the two look alike.
+     */
+    @VisibleForTesting
+    void retainRestoreCause(int restoreAttempt, String cause) {
+        String text =
+                cause == null || cause.trim().isEmpty()
+                        ? "no error was recorded for this run"
+                        : cause.trim();
+        if (text.length() > MAX_RESTORE_CAUSE_LENGTH) {
+            text = text.substring(0, MAX_RESTORE_CAUSE_LENGTH) + "... 
(truncated)";
+        }
+        restoreCauses.add(String.format("restore %s: %s", restoreAttempt, 
text));
+        while (restoreCauses.size() > MAX_RETAINED_RESTORE_CAUSES) {
+            restoreCauses.poll();
+        }
+    }
+
+    /** Why earlier runs of this pipeline ended, oldest first. Never null. */
+    public List<String> getRestoreCauses() {
+        return new ArrayList<>(restoreCauses);
+    }
+
     /**
      * reset the pipeline and task state and init state future again
      *
@@ -559,6 +610,9 @@ public class SubPlan {
         synchronized (restoreLock) {
             try {
                 pipelineRestoreNum.getAndIncrement();
+                // Record why this run ended before anything clears it: 
reset() runs just below
+                // and initStateFuture() replaces errorByPhysicalVertex 
outright.
+                retainRestoreCause(pipelineRestoreNum.get(), 
errorByPhysicalVertex.get());
                 log.info(
                         String.format(
                                 "Restore time %s, pipeline %s",
diff --git 
a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnostics.java
 
b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnostics.java
index 5acbc8307b..215e4b669f 100644
--- 
a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnostics.java
+++ 
b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnostics.java
@@ -56,6 +56,7 @@ public final class JobRuntimeDiagnostics {
     public static final String PIPELINE_STATUS = "pipelineStatus";
     public static final String RESTORE_COUNT = "restoreCount";
     public static final String MAX_RESTORE_COUNT = "maxRestoreCount";
+    public static final String RESTORE_CAUSES = "restoreCauses";
     public static final String TOTAL_PIPELINE_RESTORE_COUNT = 
"totalPipelineRestoreCount";
 
     private JobRuntimeDiagnostics() {}
@@ -90,6 +91,10 @@ public final class JobRuntimeDiagnostics {
                                     pipelineStatus == null ? null : 
pipelineStatus.toString())
                             .add(RESTORE_COUNT, restoreCount)
                             .add(MAX_RESTORE_COUNT, 
subPlan.getPipelineMaxRestoreNum())
+                            // restoreCount says a pipeline is restarting; 
this says why. The
+                            // engine keeps only the current attempt's error, 
so without this a
+                            // crash loop can be detected but not explained.
+                            .add(RESTORE_CAUSES, 
toCausesJson(subPlan.getRestoreCauses()))
                             .add(
                                     STATE_TIMESTAMPS,
                                     toTimestampsJson(
@@ -121,6 +126,15 @@ public final class JobRuntimeDiagnostics {
         }
     }
 
+    /** Causes as a JSON array, oldest first. Empty when the pipeline has not 
restarted. */
+    private static JsonArray toCausesJson(List<String> causes) {
+        JsonArray array = new JsonArray();
+        if (causes != null) {
+            causes.forEach(array::add);
+        }
+        return array;
+    }
+
     /**
      * Renders a {@code stateTimestamps} array as a state name to epoch millis 
object, skipping
      * states that were never entered.
diff --git 
a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/SubPlanTest.java
 
b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/SubPlanTest.java
index fbb400abe4..41b72a0065 100644
--- 
a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/SubPlanTest.java
+++ 
b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/SubPlanTest.java
@@ -17,7 +17,10 @@
 
 package org.apache.seatunnel.engine.server.dag.physical;
 
+import org.apache.seatunnel.api.options.EnvCommonOptions;
 import org.apache.seatunnel.engine.common.config.JobConfig;
+import org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture;
+import org.apache.seatunnel.engine.common.utils.concurrent.CompletableFuture;
 import org.apache.seatunnel.engine.core.job.JobImmutableInformation;
 import org.apache.seatunnel.engine.core.job.PipelineStatus;
 import org.apache.seatunnel.engine.server.AbstractSeaTunnelServerTest;
@@ -30,13 +33,18 @@ import org.junit.jupiter.api.Test;
 
 import com.hazelcast.map.IMap;
 
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
 import java.util.UUID;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicReference;
 
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
@@ -201,12 +209,199 @@ class SubPlanTest extends 
AbstractSeaTunnelServerTest<SubPlanTest> {
         
Assertions.assertFalse(SubPlan.isRecoverableRunningState(mockPhysicalVertex(null)));
     }
 
+    @Test
+    void shouldRetainWhyEachEarlierRunEnded() {
+        // errorByPhysicalVertex holds only the current attempt and is 
replaced on every restore,
+        // so a pipeline that has restarted repeatedly otherwise reports a 
restore count with no
+        // reason attached. These exercise the retention directly: driving a 
real restore needs a
+        // live JobMaster and checkpoint manager, which would test those 
rather than this.
+        ExecutorService executorService = Executors.newSingleThreadExecutor();
+        try {
+            SubPlan subPlan =
+                    newSubPlan(
+                            executorService,
+                            
Collections.singletonList(mockPhysicalVertex(ExecutionState.FAILED)),
+                            Collections.emptyList());
+
+            Assertions.assertTrue(subPlan.getRestoreCauses().isEmpty());
+
+            subPlan.retainRestoreCause(1, "java.net.ConnectException: 
Connection refused");
+            subPlan.retainRestoreCause(2, "java.net.ConnectException: 
Connection refused");
+
+            List<String> causes = subPlan.getRestoreCauses();
+            Assertions.assertEquals(2, causes.size());
+            Assertions.assertTrue(causes.get(0).startsWith("restore 1: "));
+            Assertions.assertTrue(causes.get(0).contains("Connection 
refused"));
+            Assertions.assertTrue(causes.get(1).startsWith("restore 2: "));
+        } finally {
+            executorService.shutdownNow();
+        }
+    }
+
+    @Test
+    void shouldRecordARestoreWithNoErrorRatherThanOmitIt() {
+        // "restarted and we do not know why" is a different and more alarming 
state than
+        // "restarted because the sink refused the connection"; dropping it 
hides the difference.
+        ExecutorService executorService = Executors.newSingleThreadExecutor();
+        try {
+            SubPlan subPlan =
+                    newSubPlan(
+                            executorService,
+                            
Collections.singletonList(mockPhysicalVertex(ExecutionState.FAILED)),
+                            Collections.emptyList());
+
+            subPlan.retainRestoreCause(1, null);
+            subPlan.retainRestoreCause(2, "   ");
+
+            List<String> causes = subPlan.getRestoreCauses();
+            Assertions.assertEquals(2, causes.size());
+            causes.forEach(
+                    cause ->
+                            Assertions.assertTrue(
+                                    cause.contains("no error was recorded"),
+                                    "unexpected placeholder: " + cause));
+        } finally {
+            executorService.shutdownNow();
+        }
+    }
+
+    @Test
+    void shouldBoundTheRetainedCausesAndTruncateLongOnes() {
+        // job.retry.times already bounds how many restores happen, but it is 
user-configurable,
+        // so the list is capped independently and each entry is truncated.
+        ExecutorService executorService = Executors.newSingleThreadExecutor();
+        try {
+            SubPlan subPlan =
+                    newSubPlan(
+                            executorService,
+                            
Collections.singletonList(mockPhysicalVertex(ExecutionState.FAILED)),
+                            Collections.emptyList());
+
+            StringBuilder huge = new StringBuilder();
+            for (int i = 0; i < 5000; i++) {
+                huge.append('x');
+            }
+            for (int attempt = 1; attempt <= 40; attempt++) {
+                subPlan.retainRestoreCause(attempt, huge.toString());
+            }
+
+            List<String> causes = subPlan.getRestoreCauses();
+            Assertions.assertEquals(16, causes.size(), "retained causes should 
be capped");
+            // The oldest were dropped, so the newest attempt must still be 
present.
+            Assertions.assertTrue(causes.get(causes.size() - 
1).startsWith("restore 40: "));
+            causes.forEach(
+                    cause ->
+                            Assertions.assertTrue(
+                                    cause.length() < 2200,
+                                    "cause was not truncated: " + 
cause.length()));
+            Assertions.assertTrue(causes.get(0).contains("(truncated)"));
+        } finally {
+            executorService.shutdownNow();
+        }
+    }
+
+    @Test
+    void shouldReturnACopyOfTheRetainedCauses() {
+        ExecutorService executorService = Executors.newSingleThreadExecutor();
+        try {
+            SubPlan subPlan =
+                    newSubPlan(
+                            executorService,
+                            
Collections.singletonList(mockPhysicalVertex(ExecutionState.FAILED)),
+                            Collections.emptyList());
+            subPlan.retainRestoreCause(1, "boom");
+
+            subPlan.getRestoreCauses().clear();
+
+            Assertions.assertEquals(1, subPlan.getRestoreCauses().size());
+        } finally {
+            executorService.shutdownNow();
+        }
+    }
+
+    @Test
+    void shouldCaptureTheRealErrorWhenARestoreIsPrepared() throws Exception {
+        // The tests above call retainRestoreCause directly, which proves the 
retention works but
+        // not that anything calls it. This drives prepareRestorePipeline 
instead.
+        //
+        // The mocked plan forwards addPipelineEndCallback to initStateFuture 
the way PhysicalPlan
+        // does, because that is the call which replaces 
errorByPhysicalVertex. With a plain mock
+        // the callback does nothing, so moving the capture below it would 
still pass here while
+        // in production every entry would become the placeholder. Wiring it 
up means this covers
+        // the reordering as well as the deletion.
+        CheckpointManager checkpointManager = mock(CheckpointManager.class);
+        PhysicalPlan physicalPlan = mock(PhysicalPlan.class);
+        JobMaster jobMaster = mock(JobMaster.class);
+        when(jobMaster.getCheckpointManager()).thenReturn(checkpointManager);
+        when(jobMaster.getPhysicalPlan()).thenReturn(physicalPlan);
+        doAnswer(
+                        invocation -> {
+                            ((SubPlan) 
invocation.getArgument(0)).initStateFuture();
+                            return null;
+                        })
+                .when(physicalPlan)
+                .addPipelineEndCallback(any(SubPlan.class));
+
+        PhysicalVertex vertex = mockPhysicalVertex(ExecutionState.FAILED);
+        when(vertex.initStateFuture())
+                .thenReturn(new PassiveCompletableFuture<>(new 
CompletableFuture<>()));
+
+        ExecutorService executorService = Executors.newSingleThreadExecutor();
+        try {
+            SubPlan subPlan =
+                    newSubPlan(
+                            executorService,
+                            Collections.singletonList(vertex),
+                            Collections.emptyList());
+            subPlan.setJobMaster(jobMaster);
+            subPlan.setCurrPipelineStatus(PipelineStatus.FAILED);
+
+            String failure = "java.sql.SQLException: Access denied for user 
'etl'@'10.0.1.7'";
+            setErrorByPhysicalVertex(subPlan, failure);
+
+            // A false return means reset() threw and the catch branch ran, 
which would let a
+            // broken restore path look like a successful capture.
+            Assertions.assertTrue(
+                    invokePrepareRestorePipeline(subPlan), 
"prepareRestorePipeline failed");
+
+            List<String> causes = subPlan.getRestoreCauses();
+            Assertions.assertEquals(1, causes.size(), "the restore cause was 
not captured");
+            Assertions.assertTrue(
+                    causes.get(0).contains("Access denied for user"),
+                    "captured the placeholder instead of the real error: " + 
causes.get(0));
+            Assertions.assertTrue(causes.get(0).startsWith("restore 1: "));
+        } finally {
+            executorService.shutdownNow();
+        }
+    }
+
+    /**
+     * Seed the field a failing PhysicalVertex would have set. Reflection is 
used only to stand in
+     * for that callback; the behaviour under test is reached through the real 
method below.
+     */
+    @SuppressWarnings("unchecked")
+    private static void setErrorByPhysicalVertex(SubPlan subPlan, String 
message) throws Exception {
+        Field field = SubPlan.class.getDeclaredField("errorByPhysicalVertex");
+        field.setAccessible(true);
+        ((AtomicReference<String>) field.get(subPlan)).set(message);
+    }
+
+    private static boolean invokePrepareRestorePipeline(SubPlan subPlan) 
throws Exception {
+        Method method = 
SubPlan.class.getDeclaredMethod("prepareRestorePipeline");
+        method.setAccessible(true);
+        return (Boolean) method.invoke(subPlan);
+    }
+
     private SubPlan newSubPlan(
             ExecutorService executorService,
             List<PhysicalVertex> physicalVertexList,
             List<PhysicalVertex> coordinatorVertexList) {
         JobConfig jobConfig = new JobConfig();
         jobConfig.setName("test");
+        // prepareRestorePipeline really sleeps for 
job.retry.interval.seconds, which defaults to
+        // 3. Only the restore test reaches that sleep, but the whole class 
shares this factory and
+        // no unit test should spend seconds waiting.
+        
jobConfig.getEnvOptions().put(EnvCommonOptions.JOB_RETRY_INTERVAL_SECONDS.key(),
 0);
 
         JobImmutableInformation jobImmutableInformation = 
mock(JobImmutableInformation.class);
         when(jobImmutableInformation.getJobId()).thenReturn(1L);
diff --git 
a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnosticsTest.java
 
b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnosticsTest.java
index 692e1fc2c5..9c80a90d89 100644
--- 
a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnosticsTest.java
+++ 
b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnosticsTest.java
@@ -39,6 +39,7 @@ import com.hazelcast.spi.impl.NodeEngineImpl;
 
 import java.util.Arrays;
 import java.util.Collections;
+import java.util.List;
 
 import static org.mockito.Mockito.doReturn;
 import static org.mockito.Mockito.doThrow;
@@ -76,13 +77,65 @@ public class JobRuntimeDiagnosticsTest {
         return timestamps;
     }
 
+    @Test
+    void testRestoreCausesAreExposedInOrder() {
+        // restoreCount says a pipeline is restarting; this says why. 
Asserting the field here is
+        // what makes removing it from JobRuntimeDiagnostics turn a test red.
+        mockPipelines(
+                mockSubPlan(
+                        1,
+                        PipelineStatus.RUNNING,
+                        2,
+                        100,
+                        Arrays.asList(
+                                "restore 1: java.net.ConnectException: 
Connection refused",
+                                "restore 2: java.sql.SQLException: Access 
denied")));
+
+        JsonObject diagnostics = JobRuntimeDiagnostics.build(server, JOB_ID);
+        JsonObject pipeline =
+                
diagnostics.get(JobRuntimeDiagnostics.PIPELINES).asArray().get(0).asObject();
+        JsonArray causes = 
pipeline.get(JobRuntimeDiagnostics.RESTORE_CAUSES).asArray();
+
+        Assertions.assertEquals(2, causes.size());
+        // Oldest first: anything reading this as a sequence depends on that 
order.
+        Assertions.assertEquals(
+                "restore 1: java.net.ConnectException: Connection refused",
+                causes.get(0).asString());
+        Assertions.assertEquals(
+                "restore 2: java.sql.SQLException: Access denied", 
causes.get(1).asString());
+    }
+
+    @Test
+    void testRestoreCausesIsAnEmptyArrayWhenNothingHasRestarted() {
+        // An empty array rather than a missing field or null, so a client can 
read it
+        // unconditionally.
+        mockPipelines(mockSubPlan(1, PipelineStatus.RUNNING, 0, 100));
+
+        JsonObject diagnostics = JobRuntimeDiagnostics.build(server, JOB_ID);
+        JsonObject pipeline =
+                
diagnostics.get(JobRuntimeDiagnostics.PIPELINES).asArray().get(0).asObject();
+
+        Assertions.assertTrue(
+                
pipeline.get(JobRuntimeDiagnostics.RESTORE_CAUSES).asArray().isEmpty());
+    }
+
     private SubPlan mockSubPlan(int pipelineId, PipelineStatus status, int 
restoreNum, int maxNum) {
+        return mockSubPlan(pipelineId, status, restoreNum, maxNum, 
Collections.emptyList());
+    }
+
+    private SubPlan mockSubPlan(
+            int pipelineId,
+            PipelineStatus status,
+            int restoreNum,
+            int maxNum,
+            List<String> restoreCauses) {
         SubPlan subPlan = mock(SubPlan.class);
         when(subPlan.getPipelineId()).thenReturn(pipelineId);
         when(subPlan.getPipelineState()).thenReturn(status);
         when(subPlan.getPipelineRestoreNum()).thenReturn(restoreNum);
         when(subPlan.getPipelineMaxRestoreNum()).thenReturn(maxNum);
         when(subPlan.getPipelineLocation()).thenReturn(new 
PipelineLocation(JOB_ID, pipelineId));
+        when(subPlan.getRestoreCauses()).thenReturn(restoreCauses);
         return subPlan;
     }
 

Reply via email to