This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13154-50cf6b6711f9316557daa3a5e17defdab4247953 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 2010a1fedbb20590ab18f69a52374478fc1e797d Author: SEZ <[email protected]> AuthorDate: Tue Nov 17 15:19:45 2026 +0000 [Improve][Zeta] Retain why each pipeline restore happened (#13154) Co-authored-by: SEZ9 <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]> --- docs/en/engines/zeta/rest-api-v2.md | 10 ++ docs/zh/engines/zeta/rest-api-v2.md | 7 + .../engine/server/dag/physical/SubPlan.java | 54 ++++++ .../server/diagnostic/JobRuntimeDiagnostics.java | 14 ++ .../engine/server/dag/physical/SubPlanTest.java | 195 +++++++++++++++++++++ .../diagnostic/JobRuntimeDiagnosticsTest.java | 53 ++++++ 6 files changed, 333 insertions(+) diff --git a/docs/en/engines/zeta/rest-api-v2.md b/docs/en/engines/zeta/rest-api-v2.md index c05f7ef575..91aef8cc5c 100644 --- a/docs/en/engines/zeta/rest-api-v2.md +++ b/docs/en/engines/zeta/rest-api-v2.md @@ -662,6 +662,10 @@ This endpoint helps troubleshoot why jobs stay in `PENDING` by showing the pendi "pipelineStatus": "RUNNING", "restoreCount": 7, "maxRestoreCount": 100, + "restoreCauses": [ + "restore 6: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[JDBC-05] ... Connection refused", + "restore 7: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[JDBC-05] ... Connection refused" + ], "stateTimestamps": { "INITIALIZING": 1755000000000, "CREATED": 1755000000200, @@ -684,6 +688,11 @@ node. It is auxiliary information: if it can not be obtained, the field is omitt failing the request. Only this endpoint returns it; `/running-jobs` does not, because collecting it for every running job would cost one more round trip to the master per job. +`diagnostics` is read from the master's in-memory job state, so like `restoreCount` it describes +this master's view of the job rather than its whole history. After a master switch the restore +counters and `restoreCauses` start again from the recovered job, and causes recorded by the +previous master are not carried over. + #### Metrics field description | Field | Description | @@ -715,6 +724,7 @@ for every running job would cost one more round trip to the master per job. | pipelines[].pipelineStatus | Current pipeline state | | pipelines[].restoreCount | How many times this pipeline has been restored since the job was submitted. A value that keeps growing while `jobStatus` stays `RUNNING` is a crash loop | | pipelines[].maxRestoreCount | Restore limit of this pipeline, from the `job.retry.times` env option | +| pipelines[].restoreCauses | Why earlier runs of this pipeline ended, oldest first, one entry per restore. The engine keeps only the current attempt's error, so without this a crash loop can be detected from `restoreCount` but not explained. Each entry is truncated, and the list is capped at the last 16 restores. A restore with no recorded error is still listed, because "restarted for an unknown reason" is a different state from "restarted because the sink refused the connection" | | pipelines[].stateTimestamps | Epoch millis when the pipeline entered each state. After a restore, the timestamps of the new attempt overwrite the previous ones | | totalPipelineRestoreCount | Sum of `restoreCount` over all pipelines of the job | diff --git a/docs/zh/engines/zeta/rest-api-v2.md b/docs/zh/engines/zeta/rest-api-v2.md index 910b90d78f..b233c47a12 100644 --- a/docs/zh/engines/zeta/rest-api-v2.md +++ b/docs/zh/engines/zeta/rest-api-v2.md @@ -653,6 +653,10 @@ seatunnel: "pipelineStatus": "RUNNING", "restoreCount": 7, "maxRestoreCount": 100, + "restoreCauses": [ + "restore 6: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[JDBC-05] ... Connection refused", + "restore 7: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[JDBC-05] ... Connection refused" + ], "stateTimestamps": { "INITIALIZING": 1755000000000, "CREATED": 1755000000200, @@ -672,6 +676,8 @@ seatunnel: `finishTime`, `errorMsg` 字段在Job结束时会返回,结束状态为不为RUNNING,可能为FINISHED,可能为CANCEL `diagnostics` 字段在Job运行中且能从Master节点读取到诊断信息时返回。它属于辅助信息,读取失败时该字段会被省略,不会导致请求失败。该字段只在本接口返回,`/running-jobs` 不返回:为列表中的每个Job收集诊断信息会额外增加一次到Master的请求。 +`diagnostics` 读取的是 Master 内存中的 Job 状态,因此与 `restoreCount` 一样,它描述的是当前这个 Master 对该 Job 的视图,而不是完整历史。Master 切换后,恢复计数与 `restoreCauses` 会从被恢复的 Job 重新开始,上一个 Master 记录的原因不会延续。 + #### 指标字段说明 | 字段 | 说明 | @@ -703,6 +709,7 @@ seatunnel: | pipelines[].pipelineStatus | Pipeline 当前状态 | | pipelines[].restoreCount | 自 Job 提交以来该 Pipeline 被恢复(重启)的次数。若 `jobStatus` 一直是 `RUNNING` 而该值不断增长,说明处于崩溃重启循环中 | | pipelines[].maxRestoreCount | 该 Pipeline 的恢复次数上限,来自 `job.retry.times` env 配置 | +| pipelines[].restoreCauses | 该 Pipeline 此前每一次运行结束的原因,按时间从旧到新,每次恢复一条。引擎只保留当前这次尝试的错误,因此若没有该字段,仅凭 `restoreCount` 只能发现崩溃重启循环、无法知道原因。每条都会截断,且只保留最近 16 次恢复。未记录到错误的那次恢复也会列出,因为「因未知原因重启」与「因 Sink 拒绝连接而重启」是两种不同的状态 | | pipelines[].stateTimestamps | Pipeline 进入各状态的时间戳(毫秒)。发生恢复后,新一轮的时间戳会覆盖上一轮 | | totalPipelineRestoreCount | Job 下所有 Pipeline 的 `restoreCount` 之和 | diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java index c4ee893ad4..008d2314da 100644 --- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java +++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java @@ -43,8 +43,11 @@ import lombok.Data; import lombok.NonNull; import lombok.extern.slf4j.Slf4j; +import java.util.ArrayList; import java.util.List; import java.util.Map; +import java.util.Queue; +import java.util.concurrent.ConcurrentLinkedQueue; import java.util.concurrent.ExecutorService; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; @@ -94,6 +97,27 @@ public class SubPlan { /** The error throw by physicalVertex, should be set when physicalVertex throw error. */ private AtomicReference<String> errorByPhysicalVertex = new AtomicReference<>(); + /** + * Why each earlier run of this pipeline ended, oldest first. + * + * <p>{@link #errorByPhysicalVertex} only holds the current attempt and is replaced on every + * restore, so by the time anyone looks at a pipeline that has restarted repeatedly the reasons + * are gone. The restore counter then says a crash loop is happening without saying what is + * causing it, which is the first thing an operator needs. + * + * <p>The size is bounded twice over: a restore only happens while {@code pipelineRestoreNum < + * pipelineMaxRestoreNum}, which {@code job.retry.times} sets and defaults to 3, and {@link + * #MAX_RETAINED_RESTORE_CAUSES} caps it regardless in case that is configured high. Each entry + * is truncated, so the whole list stays small enough to live beside the pipeline. + */ + private final Queue<String> restoreCauses = new ConcurrentLinkedQueue<>(); + + /** Upper bound on {@link #restoreCauses}, independent of {@code job.retry.times}. */ + private static final int MAX_RETAINED_RESTORE_CAUSES = 16; + + /** Longest retained cause. Enough for a message and the first frames that matter. */ + private static final int MAX_RESTORE_CAUSE_LENGTH = 2048; + private final ExecutorService executorService; private JobMaster jobMaster; @@ -550,6 +574,33 @@ public class SubPlan { Constant.OPERATION_RETRY_SLEEP)); } + /** + * Keep why a run ended, dropping the oldest when full. + * + * <p>A restore with no recorded error is still worth a placeholder: "restarted and we do not + * know why" is a different and more alarming state than "restarted because the sink refused the + * connection", and silently omitting it would make the two look alike. + */ + @VisibleForTesting + void retainRestoreCause(int restoreAttempt, String cause) { + String text = + cause == null || cause.trim().isEmpty() + ? "no error was recorded for this run" + : cause.trim(); + if (text.length() > MAX_RESTORE_CAUSE_LENGTH) { + text = text.substring(0, MAX_RESTORE_CAUSE_LENGTH) + "... (truncated)"; + } + restoreCauses.add(String.format("restore %s: %s", restoreAttempt, text)); + while (restoreCauses.size() > MAX_RETAINED_RESTORE_CAUSES) { + restoreCauses.poll(); + } + } + + /** Why earlier runs of this pipeline ended, oldest first. Never null. */ + public List<String> getRestoreCauses() { + return new ArrayList<>(restoreCauses); + } + /** * reset the pipeline and task state and init state future again * @@ -559,6 +610,9 @@ public class SubPlan { synchronized (restoreLock) { try { pipelineRestoreNum.getAndIncrement(); + // Record why this run ended before anything clears it: reset() runs just below + // and initStateFuture() replaces errorByPhysicalVertex outright. + retainRestoreCause(pipelineRestoreNum.get(), errorByPhysicalVertex.get()); log.info( String.format( "Restore time %s, pipeline %s", diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnostics.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnostics.java index 5acbc8307b..215e4b669f 100644 --- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnostics.java +++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnostics.java @@ -56,6 +56,7 @@ public final class JobRuntimeDiagnostics { public static final String PIPELINE_STATUS = "pipelineStatus"; public static final String RESTORE_COUNT = "restoreCount"; public static final String MAX_RESTORE_COUNT = "maxRestoreCount"; + public static final String RESTORE_CAUSES = "restoreCauses"; public static final String TOTAL_PIPELINE_RESTORE_COUNT = "totalPipelineRestoreCount"; private JobRuntimeDiagnostics() {} @@ -90,6 +91,10 @@ public final class JobRuntimeDiagnostics { pipelineStatus == null ? null : pipelineStatus.toString()) .add(RESTORE_COUNT, restoreCount) .add(MAX_RESTORE_COUNT, subPlan.getPipelineMaxRestoreNum()) + // restoreCount says a pipeline is restarting; this says why. The + // engine keeps only the current attempt's error, so without this a + // crash loop can be detected but not explained. + .add(RESTORE_CAUSES, toCausesJson(subPlan.getRestoreCauses())) .add( STATE_TIMESTAMPS, toTimestampsJson( @@ -121,6 +126,15 @@ public final class JobRuntimeDiagnostics { } } + /** Causes as a JSON array, oldest first. Empty when the pipeline has not restarted. */ + private static JsonArray toCausesJson(List<String> causes) { + JsonArray array = new JsonArray(); + if (causes != null) { + causes.forEach(array::add); + } + return array; + } + /** * Renders a {@code stateTimestamps} array as a state name to epoch millis object, skipping * states that were never entered. diff --git a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/SubPlanTest.java b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/SubPlanTest.java index fbb400abe4..41b72a0065 100644 --- a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/SubPlanTest.java +++ b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/SubPlanTest.java @@ -17,7 +17,10 @@ package org.apache.seatunnel.engine.server.dag.physical; +import org.apache.seatunnel.api.options.EnvCommonOptions; import org.apache.seatunnel.engine.common.config.JobConfig; +import org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture; +import org.apache.seatunnel.engine.common.utils.concurrent.CompletableFuture; import org.apache.seatunnel.engine.core.job.JobImmutableInformation; import org.apache.seatunnel.engine.core.job.PipelineStatus; import org.apache.seatunnel.engine.server.AbstractSeaTunnelServerTest; @@ -30,13 +33,18 @@ import org.junit.jupiter.api.Test; import com.hazelcast.map.IMap; +import java.lang.reflect.Field; +import java.lang.reflect.Method; import java.util.Arrays; import java.util.Collections; import java.util.List; import java.util.UUID; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicReference; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doAnswer; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.times; import static org.mockito.Mockito.verify; @@ -201,12 +209,199 @@ class SubPlanTest extends AbstractSeaTunnelServerTest<SubPlanTest> { Assertions.assertFalse(SubPlan.isRecoverableRunningState(mockPhysicalVertex(null))); } + @Test + void shouldRetainWhyEachEarlierRunEnded() { + // errorByPhysicalVertex holds only the current attempt and is replaced on every restore, + // so a pipeline that has restarted repeatedly otherwise reports a restore count with no + // reason attached. These exercise the retention directly: driving a real restore needs a + // live JobMaster and checkpoint manager, which would test those rather than this. + ExecutorService executorService = Executors.newSingleThreadExecutor(); + try { + SubPlan subPlan = + newSubPlan( + executorService, + Collections.singletonList(mockPhysicalVertex(ExecutionState.FAILED)), + Collections.emptyList()); + + Assertions.assertTrue(subPlan.getRestoreCauses().isEmpty()); + + subPlan.retainRestoreCause(1, "java.net.ConnectException: Connection refused"); + subPlan.retainRestoreCause(2, "java.net.ConnectException: Connection refused"); + + List<String> causes = subPlan.getRestoreCauses(); + Assertions.assertEquals(2, causes.size()); + Assertions.assertTrue(causes.get(0).startsWith("restore 1: ")); + Assertions.assertTrue(causes.get(0).contains("Connection refused")); + Assertions.assertTrue(causes.get(1).startsWith("restore 2: ")); + } finally { + executorService.shutdownNow(); + } + } + + @Test + void shouldRecordARestoreWithNoErrorRatherThanOmitIt() { + // "restarted and we do not know why" is a different and more alarming state than + // "restarted because the sink refused the connection"; dropping it hides the difference. + ExecutorService executorService = Executors.newSingleThreadExecutor(); + try { + SubPlan subPlan = + newSubPlan( + executorService, + Collections.singletonList(mockPhysicalVertex(ExecutionState.FAILED)), + Collections.emptyList()); + + subPlan.retainRestoreCause(1, null); + subPlan.retainRestoreCause(2, " "); + + List<String> causes = subPlan.getRestoreCauses(); + Assertions.assertEquals(2, causes.size()); + causes.forEach( + cause -> + Assertions.assertTrue( + cause.contains("no error was recorded"), + "unexpected placeholder: " + cause)); + } finally { + executorService.shutdownNow(); + } + } + + @Test + void shouldBoundTheRetainedCausesAndTruncateLongOnes() { + // job.retry.times already bounds how many restores happen, but it is user-configurable, + // so the list is capped independently and each entry is truncated. + ExecutorService executorService = Executors.newSingleThreadExecutor(); + try { + SubPlan subPlan = + newSubPlan( + executorService, + Collections.singletonList(mockPhysicalVertex(ExecutionState.FAILED)), + Collections.emptyList()); + + StringBuilder huge = new StringBuilder(); + for (int i = 0; i < 5000; i++) { + huge.append('x'); + } + for (int attempt = 1; attempt <= 40; attempt++) { + subPlan.retainRestoreCause(attempt, huge.toString()); + } + + List<String> causes = subPlan.getRestoreCauses(); + Assertions.assertEquals(16, causes.size(), "retained causes should be capped"); + // The oldest were dropped, so the newest attempt must still be present. + Assertions.assertTrue(causes.get(causes.size() - 1).startsWith("restore 40: ")); + causes.forEach( + cause -> + Assertions.assertTrue( + cause.length() < 2200, + "cause was not truncated: " + cause.length())); + Assertions.assertTrue(causes.get(0).contains("(truncated)")); + } finally { + executorService.shutdownNow(); + } + } + + @Test + void shouldReturnACopyOfTheRetainedCauses() { + ExecutorService executorService = Executors.newSingleThreadExecutor(); + try { + SubPlan subPlan = + newSubPlan( + executorService, + Collections.singletonList(mockPhysicalVertex(ExecutionState.FAILED)), + Collections.emptyList()); + subPlan.retainRestoreCause(1, "boom"); + + subPlan.getRestoreCauses().clear(); + + Assertions.assertEquals(1, subPlan.getRestoreCauses().size()); + } finally { + executorService.shutdownNow(); + } + } + + @Test + void shouldCaptureTheRealErrorWhenARestoreIsPrepared() throws Exception { + // The tests above call retainRestoreCause directly, which proves the retention works but + // not that anything calls it. This drives prepareRestorePipeline instead. + // + // The mocked plan forwards addPipelineEndCallback to initStateFuture the way PhysicalPlan + // does, because that is the call which replaces errorByPhysicalVertex. With a plain mock + // the callback does nothing, so moving the capture below it would still pass here while + // in production every entry would become the placeholder. Wiring it up means this covers + // the reordering as well as the deletion. + CheckpointManager checkpointManager = mock(CheckpointManager.class); + PhysicalPlan physicalPlan = mock(PhysicalPlan.class); + JobMaster jobMaster = mock(JobMaster.class); + when(jobMaster.getCheckpointManager()).thenReturn(checkpointManager); + when(jobMaster.getPhysicalPlan()).thenReturn(physicalPlan); + doAnswer( + invocation -> { + ((SubPlan) invocation.getArgument(0)).initStateFuture(); + return null; + }) + .when(physicalPlan) + .addPipelineEndCallback(any(SubPlan.class)); + + PhysicalVertex vertex = mockPhysicalVertex(ExecutionState.FAILED); + when(vertex.initStateFuture()) + .thenReturn(new PassiveCompletableFuture<>(new CompletableFuture<>())); + + ExecutorService executorService = Executors.newSingleThreadExecutor(); + try { + SubPlan subPlan = + newSubPlan( + executorService, + Collections.singletonList(vertex), + Collections.emptyList()); + subPlan.setJobMaster(jobMaster); + subPlan.setCurrPipelineStatus(PipelineStatus.FAILED); + + String failure = "java.sql.SQLException: Access denied for user 'etl'@'10.0.1.7'"; + setErrorByPhysicalVertex(subPlan, failure); + + // A false return means reset() threw and the catch branch ran, which would let a + // broken restore path look like a successful capture. + Assertions.assertTrue( + invokePrepareRestorePipeline(subPlan), "prepareRestorePipeline failed"); + + List<String> causes = subPlan.getRestoreCauses(); + Assertions.assertEquals(1, causes.size(), "the restore cause was not captured"); + Assertions.assertTrue( + causes.get(0).contains("Access denied for user"), + "captured the placeholder instead of the real error: " + causes.get(0)); + Assertions.assertTrue(causes.get(0).startsWith("restore 1: ")); + } finally { + executorService.shutdownNow(); + } + } + + /** + * Seed the field a failing PhysicalVertex would have set. Reflection is used only to stand in + * for that callback; the behaviour under test is reached through the real method below. + */ + @SuppressWarnings("unchecked") + private static void setErrorByPhysicalVertex(SubPlan subPlan, String message) throws Exception { + Field field = SubPlan.class.getDeclaredField("errorByPhysicalVertex"); + field.setAccessible(true); + ((AtomicReference<String>) field.get(subPlan)).set(message); + } + + private static boolean invokePrepareRestorePipeline(SubPlan subPlan) throws Exception { + Method method = SubPlan.class.getDeclaredMethod("prepareRestorePipeline"); + method.setAccessible(true); + return (Boolean) method.invoke(subPlan); + } + private SubPlan newSubPlan( ExecutorService executorService, List<PhysicalVertex> physicalVertexList, List<PhysicalVertex> coordinatorVertexList) { JobConfig jobConfig = new JobConfig(); jobConfig.setName("test"); + // prepareRestorePipeline really sleeps for job.retry.interval.seconds, which defaults to + // 3. Only the restore test reaches that sleep, but the whole class shares this factory and + // no unit test should spend seconds waiting. + jobConfig.getEnvOptions().put(EnvCommonOptions.JOB_RETRY_INTERVAL_SECONDS.key(), 0); JobImmutableInformation jobImmutableInformation = mock(JobImmutableInformation.class); when(jobImmutableInformation.getJobId()).thenReturn(1L); diff --git a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnosticsTest.java b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnosticsTest.java index 692e1fc2c5..9c80a90d89 100644 --- a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnosticsTest.java +++ b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/diagnostic/JobRuntimeDiagnosticsTest.java @@ -39,6 +39,7 @@ import com.hazelcast.spi.impl.NodeEngineImpl; import java.util.Arrays; import java.util.Collections; +import java.util.List; import static org.mockito.Mockito.doReturn; import static org.mockito.Mockito.doThrow; @@ -76,13 +77,65 @@ public class JobRuntimeDiagnosticsTest { return timestamps; } + @Test + void testRestoreCausesAreExposedInOrder() { + // restoreCount says a pipeline is restarting; this says why. Asserting the field here is + // what makes removing it from JobRuntimeDiagnostics turn a test red. + mockPipelines( + mockSubPlan( + 1, + PipelineStatus.RUNNING, + 2, + 100, + Arrays.asList( + "restore 1: java.net.ConnectException: Connection refused", + "restore 2: java.sql.SQLException: Access denied"))); + + JsonObject diagnostics = JobRuntimeDiagnostics.build(server, JOB_ID); + JsonObject pipeline = + diagnostics.get(JobRuntimeDiagnostics.PIPELINES).asArray().get(0).asObject(); + JsonArray causes = pipeline.get(JobRuntimeDiagnostics.RESTORE_CAUSES).asArray(); + + Assertions.assertEquals(2, causes.size()); + // Oldest first: anything reading this as a sequence depends on that order. + Assertions.assertEquals( + "restore 1: java.net.ConnectException: Connection refused", + causes.get(0).asString()); + Assertions.assertEquals( + "restore 2: java.sql.SQLException: Access denied", causes.get(1).asString()); + } + + @Test + void testRestoreCausesIsAnEmptyArrayWhenNothingHasRestarted() { + // An empty array rather than a missing field or null, so a client can read it + // unconditionally. + mockPipelines(mockSubPlan(1, PipelineStatus.RUNNING, 0, 100)); + + JsonObject diagnostics = JobRuntimeDiagnostics.build(server, JOB_ID); + JsonObject pipeline = + diagnostics.get(JobRuntimeDiagnostics.PIPELINES).asArray().get(0).asObject(); + + Assertions.assertTrue( + pipeline.get(JobRuntimeDiagnostics.RESTORE_CAUSES).asArray().isEmpty()); + } + private SubPlan mockSubPlan(int pipelineId, PipelineStatus status, int restoreNum, int maxNum) { + return mockSubPlan(pipelineId, status, restoreNum, maxNum, Collections.emptyList()); + } + + private SubPlan mockSubPlan( + int pipelineId, + PipelineStatus status, + int restoreNum, + int maxNum, + List<String> restoreCauses) { SubPlan subPlan = mock(SubPlan.class); when(subPlan.getPipelineId()).thenReturn(pipelineId); when(subPlan.getPipelineState()).thenReturn(status); when(subPlan.getPipelineRestoreNum()).thenReturn(restoreNum); when(subPlan.getPipelineMaxRestoreNum()).thenReturn(maxNum); when(subPlan.getPipelineLocation()).thenReturn(new PipelineLocation(JOB_ID, pipelineId)); + when(subPlan.getRestoreCauses()).thenReturn(restoreCauses); return subPlan; }
