This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new be914d2c01 [Docs][Zeta] Add runtime execution graph design (#11612)
be914d2c01 is described below
commit be914d2c019c5e72cfea0f5d4f699648e8b993a9
Author: danielnadean <[email protected]>
AuthorDate: Sun Aug 23 08:58:50 2026 +0800
[Docs][Zeta] Add runtime execution graph design (#11612)
Co-authored-by: DanielLeens <[email protected]>
---
docs/en/engines/zeta/busyness-and-backpressure.md | 1 +
docs/en/engines/zeta/runtime-execution-graph.md | 188 ++++++++++++++++++++++
docs/en/engines/zeta/web-ui.md | 3 +
docs/sidebars.js | 1 +
docs/zh/engines/zeta/busyness-and-backpressure.md | 1 +
docs/zh/engines/zeta/runtime-execution-graph.md | 188 ++++++++++++++++++++++
docs/zh/engines/zeta/web-ui.md | 3 +
7 files changed, 385 insertions(+)
diff --git a/docs/en/engines/zeta/busyness-and-backpressure.md
b/docs/en/engines/zeta/busyness-and-backpressure.md
index 0bc503ed0f..b2c49c1cc8 100644
--- a/docs/en/engines/zeta/busyness-and-backpressure.md
+++ b/docs/en/engines/zeta/busyness-and-backpressure.md
@@ -220,6 +220,7 @@ curl
"http://<master-host>:8080/metrics/realtime/jobs/<jobId>/vertices?windowMs=
## Related Documentation
+- [Runtime Execution Graph](./runtime-execution-graph.md)
- [REST API and Web UI](./rest-api-and-web-ui.md)
- [Web UI](./web-ui.md)
- [RESTful API V2](./rest-api-v2.md)
diff --git a/docs/en/engines/zeta/runtime-execution-graph.md
b/docs/en/engines/zeta/runtime-execution-graph.md
new file mode 100644
index 0000000000..5b6908ed38
--- /dev/null
+++ b/docs/en/engines/zeta/runtime-execution-graph.md
@@ -0,0 +1,188 @@
+# Runtime Execution Graph
+
+## Status
+
+This page is the proposed design contract for [issue
#11351](https://github.com/apache/seatunnel/issues/11351). The first delivery
should turn the existing Job Detail DAG and realtime observability data into a
focused runtime diagnosis graph without adding another metrics pipeline.
+
+The design is intentionally bounded:
+
+- reuse the canonical Zeta job DAG as the graph topology
+- reuse active-master in-memory realtime metrics for node and edge health
+- keep drill-down data in the existing metric APIs, checkpoint REST endpoints,
and UI detail panels
+- avoid persistent metric snapshots and long-term replay in the first version
+
+## Problem
+
+SeaTunnel already exposes job topology, vertex metrics, edge metrics,
checkpoint history, exceptions, and logs. Operators still need a single path
that answers these questions while a job is running:
+
+- which vertex is busy, idle, or waiting right now
+- which queue-backed edge is blocked or filling up
+- whether the current bottleneck is closer to Source read, Transform
processing, Sink write, checkpoint or commit work, or an external system
+- where a slowdown appears to start before checking tables and logs
+
+A runtime execution graph should be the diagnosis entry point. It should not
become a second monitoring system.
+
+## Existing Foundation
+
+The V1 runtime graph should compose existing Zeta contracts:
+
+| Area | Existing contract |
+|---|---|
+| Topology | Job Detail already renders the `JobDAGInfo` topology. |
+| Vertex metrics | `/metrics/realtime/jobs/{jobId}/vertices` returns recent
Source, Transform, and Sink buckets. |
+| Edge metrics | `/metrics/realtime/jobs/{jobId}/edges` returns queue-backed
edge buckets with `queueId` and `targetVertexId`. |
+| Checkpoint state | `/jobs/checkpoints/:jobId` and
`/jobs/checkpoints/history/:jobId` expose checkpoint overview and history. |
+| Error and logs | Job detail, exception, and log APIs already provide failure
context. |
+
+The active master keeps only a short in-memory window. The current collector
polls worker metrics every 5 seconds, REST query windows default to 3 minutes,
and the maximum window is 10 minutes.
+
+## Goals
+
+1. Show node health directly on the DAG.
+2. Show queue-backed edge health directly on the DAG.
+3. Make the hottest vertex and most blocked edge visible without switching
pages.
+4. Let users drill down from a graph object to the existing metric table or
detail drawer.
+5. Surface checkpoint and task error context near the graph without making
them graph-only contracts.
+6. Keep large DAG behavior predictable and cheaper than rendering every signal
on every element.
+
+## Non-Goals
+
+- distributed tracing
+- arbitrary historical replay
+- persistent runtime metric snapshots
+- connector-specific graph widgets
+- a new backend metrics data model parallel to realtime observability
+- automatic root cause judgement that replaces operator validation
+
+## Runtime Graph Data Model
+
+### Topology
+
+The graph topology is the current job DAG:
+
+- `jobId`
+- `vertexId`
+- vertex type, such as Source, Transform, or Sink
+- directed edges between vertices
+- edge metadata that can be correlated with `targetVertexId`
+
+The runtime graph must not invent a separate topology. If the execution DAG
changes in a future version, the graph should follow the engine DAG contract
rather than maintaining its own shape.
+
+### Node Runtime State
+
+Each graph node should merge the latest realtime vertex point by `vertexId`.
+
+| Vertex type | Primary visual signal | Supporting fields |
+|---|---|---|
+| Source | `sourceReadRatio` and `sourceIdleRatio` | `sourceReadNs`,
`sourceIdleNs`, `subtaskCount` |
+| Transform | `transformBusyRatio` | `transformProcessNsPerRecord`,
`transformRecordsIn`, `transformRecordsOut`, `subtaskCount` |
+| Sink | `sinkBusyRatio` | `sinkWriteNsPerRecord`, `sinkRecordsIn`,
`sinkPrepareCommitNs`, `sinkCommitNs`, `sinkAbortNs`, `subtaskCount` |
+
+The primary node color should represent the metric that best matches the
vertex type. For example, Source uses read or idle ratios, Transform uses
transform busy ratio, and Sink uses sink busy ratio.
+
+### Edge Runtime State
+
+Only queue-backed edges can expose backpressure metrics in V1. Each graph edge
should merge the latest realtime edge point by `targetVertexId` when the REST
response provides it, or by decoding `queueId` when necessary.
+
+| Field | Meaning |
+|---|---|
+| `queueId` | Stable queue metric identifier for realtime aggregation. |
+| `targetVertexId` | Downstream vertex used to map a queue metric back to the
DAG edge. |
+| `bpRatio` | Producer wait time ratio in the bucket. |
+| `queueFillRatio` | Latest sampled queue occupancy ratio. |
+| `queueSize` | Latest sampled queue size. |
+| `queueCapacity` | Latest sampled queue capacity. |
+
+Edge color should represent `bpRatio`. Edge width should represent
`queueFillRatio`. The detail drawer should show the raw fields and recent
bucket series.
+
+## Stable Contract and Best-Effort Signals
+
+The graph needs to distinguish stable identifiers from diagnostic signals that
are useful but sampled.
+
+Stable contract fields:
+
+- `jobId`
+- `vertexId`
+- `queueId`
+- `targetVertexId`
+- `bucketMs`
+- `fromMs`
+- `toMs`
+- point timestamp `ts`
+- `subtaskCount`
+
+Best-effort diagnostic fields:
+
+- busy ratios
+- idle ratios
+- per-record time estimates
+- queue size
+- queue fill ratio
+- producer wait ratio
+- checkpoint and error summary badges
+
+Best-effort fields may fluctuate around recovery, rescaling, counter resets,
or sampling delay. The UI should present them as live diagnosis signals, not as
audited accounting values.
+
+## Refresh, Retention, and Cost
+
+V1 should keep the existing refresh and retention model:
+
+- worker counters are collected by the active master
+- collection remains a short in-memory window
+- default REST query window is 3 minutes
+- maximum REST query window is 10 minutes
+- UI refresh can be more frequent than the collector interval, but must
tolerate repeated buckets
+- no runtime graph data is written to disk by default
+
+This keeps runtime graph cost proportional to the existing realtime
observability feature instead of adding another polling or persistence loop.
+
+## Checkpoint and Error Context
+
+Checkpoint and error signals should be visible near the graph, but they should
remain separate contracts in V1:
+
+- checkpoint overview and history continue to come from checkpoint REST
endpoints
+- job exceptions and logs continue to come from existing job detail APIs
+- the graph may show a small status badge or link, but drill-down should open
the existing detail panel
+
+This avoids mixing checkpoint lifecycle data into the realtime metrics
endpoint before the lifecycle and retention rules are explicitly agreed.
+
+## Large DAG Degradation
+
+Large DAGs can become unreadable and expensive to repaint. V1 should degrade
instead of trying to render every signal at full detail.
+
+Recommended behavior:
+
+- keep topology rendering available
+- show a summarized health table with the hottest vertices and most blocked
edges
+- limit automatic graph refit and animation once the DAG is large
+- keep drill-down available for selected vertices and edges
+- avoid increasing master collection frequency to compensate for UI complexity
+
+The implementation PR should document the chosen size threshold and keep it as
a UI rendering rule, not a backend sampling rule.
+
+## V1 Delivery Plan
+
+1. Keep the active-master realtime REST endpoints as the source of vertex and
edge runtime state.
+2. Render node color from the latest vertex point for the matching `vertexId`.
+3. Render edge color and width from the latest edge point for the matching
`targetVertexId`.
+4. Show recent point series and raw fields in the existing detail drawer.
+5. Add checkpoint and error entry points as nearby context instead of
embedding new fields in realtime metrics.
+6. Add a large-DAG fallback that lists hottest vertices and most blocked edges
when full visual overlay is too noisy.
+7. Update REST, Web UI, and operational documentation in English and Chinese.
+
+## Validation Requirements
+
+Implementation should be accepted only when these checks are covered:
+
+- backend realtime edge and vertex response tests cover current fields and
`targetVertexId` mapping
+- UI tests cover node coloring, edge coloring, edge width, and disabled
metrics behavior
+- large-DAG fallback is tested with a deterministic synthetic DAG
+- docs state that realtime windows are in-memory and best effort
+- no new persistent metric table, file, or write path is introduced for V1
+
+## Related Documentation
+
+- [Realtime Observability](./realtime-observability.md)
+- [Busyness and Backpressure](./busyness-and-backpressure.md)
+- [Web UI](./web-ui.md)
+- [RESTful API V2](./rest-api-v2.md)
diff --git a/docs/en/engines/zeta/web-ui.md b/docs/en/engines/zeta/web-ui.md
index 33ffdb8e67..944d6fa299 100644
--- a/docs/en/engines/zeta/web-ui.md
+++ b/docs/en/engines/zeta/web-ui.md
@@ -77,6 +77,8 @@ On the Job Detail page, the DAG view can display realtime
metrics for the recent
This capability requires the job to enable `env.engine.observability` or
configure an option that auto-enables it, such as `async_boundaries` or
`split_sink_io`. See [Realtime Observability](realtime-observability.md) for
configuration and metric semantics.
+For the runtime graph design boundary and large-DAG fallback rules, see
[Runtime Execution Graph](runtime-execution-graph.md).
+
### Finished Jobs
The "Finished Jobs" section displays jobs that have reached a terminal state,
such as finished, failed, cancelled, or savepoint done. Users can review
historical records and open the detail page to inspect configuration, exception
text, metrics retained by the engine, and logs.
@@ -103,5 +105,6 @@ The "Master" section displays system monitoring information
for master nodes. Us
- [REST API and Web UI](./rest-api-and-web-ui.md)
- [REST API V2](./rest-api-v2.md)
+- [Runtime Execution Graph](./runtime-execution-graph.md)
- [Job Lifecycle API](./rest-api-job-lifecycle.md)
- [Security](./security.md)
diff --git a/docs/sidebars.js b/docs/sidebars.js
index 2f8c47ae51..cd737212fb 100644
--- a/docs/sidebars.js
+++ b/docs/sidebars.js
@@ -272,6 +272,7 @@ const sidebars = {
]
},
"engines/zeta/web-ui",
+ "engines/zeta/runtime-execution-graph",
"engines/zeta/security",
"engines/zeta/python-sdk"
]
diff --git a/docs/zh/engines/zeta/busyness-and-backpressure.md
b/docs/zh/engines/zeta/busyness-and-backpressure.md
index c4151f16bd..7ecbe7139c 100644
--- a/docs/zh/engines/zeta/busyness-and-backpressure.md
+++ b/docs/zh/engines/zeta/busyness-and-backpressure.md
@@ -220,6 +220,7 @@ curl
"http://<master-host>:8080/metrics/realtime/jobs/<jobId>/vertices?windowMs=
## 相关文档
+- [运行时执行图](./runtime-execution-graph.md)
- [REST API 与 Web UI](./rest-api-and-web-ui.md)
- [Web UI](./web-ui.md)
- [RESTful API V2](./rest-api-v2.md)
diff --git a/docs/zh/engines/zeta/runtime-execution-graph.md
b/docs/zh/engines/zeta/runtime-execution-graph.md
new file mode 100644
index 0000000000..b44663cf8f
--- /dev/null
+++ b/docs/zh/engines/zeta/runtime-execution-graph.md
@@ -0,0 +1,188 @@
+# 运行时执行图
+
+## 状态
+
+本文是 [issue #11351](https://github.com/apache/seatunnel/issues/11351)
的设计契约草案。第一阶段交付应当把现有 Job Detail DAG 和实时可观测指标组合成一个聚焦诊断的运行时执行图，而不是新增另一条指标链路。
+
+该设计刻意控制边界：
+
+- 复用 Zeta 作业 DAG 作为唯一图拓扑
+- 复用 Active Master 内存中的实时指标窗口展示节点和边的健康状态
+- 复用现有指标 API、checkpoint REST 接口和 UI 详情面板做 drill-down
+- 第一版不引入持久化指标快照，也不提供长期回放
+
+## 问题
+
+SeaTunnel 已经暴露了作业拓扑、vertex 指标、edge 指标、checkpoint
历史、异常和日志。运行中的作业出现变慢或阻塞时，使用者仍然需要一条统一路径回答这些问题：
+
+- 哪个 vertex 现在忙、空闲或等待
+- 哪条带队列的 edge 正在阻塞或逐渐填满
+- 当前瓶颈更接近 Source 读取、Transform 处理、Sink 写入、checkpoint 或 commit，还是外部系统
+- 在进入表格和日志前，先判断 slowdown 大概率从哪里开始
+
+运行时执行图应当作为诊断入口，而不是变成第二套监控系统。
+
+## 现有基础
+
+V1 运行时执行图应组合现有 Zeta 契约：
+
+| 范围 | 现有契约 |
+|---|---|
+| 拓扑 | Job Detail 已经渲染 `JobDAGInfo` 拓扑。 |
+| Vertex 指标 | `/metrics/realtime/jobs/{jobId}/vertices` 返回最近窗口内
Source、Transform、Sink 的 bucket。 |
+| Edge 指标 | `/metrics/realtime/jobs/{jobId}/edges` 返回带 `queueId` 与
`targetVertexId` 的队列 edge bucket。 |
+| Checkpoint 状态 | `/jobs/checkpoints/:jobId` 和
`/jobs/checkpoints/history/:jobId` 暴露 checkpoint 概览与历史。 |
+| 异常与日志 | Job Detail、Exception 和 Log API 已经提供失败上下文。 |
+
+Active Master 只保留短窗口内存数据。当前 collector 每 5 秒拉取一次 worker 指标，REST 查询窗口默认 3 分钟，最大
10 分钟。
+
+## 目标
+
+1. 在 DAG 上直接展示节点健康状态。
+2. 在 DAG 上直接展示队列 edge 健康状态。
+3. 不切换页面也能看出最热 vertex 和最阻塞 edge。
+4. 点击图上的节点或边，可以进入现有指标表格或详情抽屉。
+5. 在图附近展示 checkpoint 与任务错误上下文，但不把它们变成 graph-only 契约。
+6. 对大 DAG 提供可预期、成本更低的降级展示。
+
+## 非目标
+
+- 分布式 tracing
+- 任意时间范围的历史回放
+- 持久化运行时指标快照
+- connector 专属图组件
+- 与 realtime observability 并行的新后端指标模型
+- 替代人工判断的自动根因结论
+
+## 运行时图数据模型
+
+### 拓扑
+
+图拓扑来自当前作业 DAG：
+
+- `jobId`
+- `vertexId`
+- vertex 类型，例如 Source、Transform 或 Sink
+- vertex 之间的有向边
+- 可与 `targetVertexId` 关联的 edge 元数据
+
+运行时执行图不能发明另一套拓扑。如果未来执行 DAG 发生变化，图应跟随引擎 DAG 契约，而不是维护自己的形状。
+
+### 节点运行状态
+
+每个图节点通过 `vertexId` 合并最近一个 realtime vertex point。
+
+| Vertex 类型 | 主要视觉信号 | 辅助字段 |
+|---|---|---|
+| Source | `sourceReadRatio` 和 `sourceIdleRatio` |
`sourceReadNs`、`sourceIdleNs`、`subtaskCount` |
+| Transform | `transformBusyRatio` |
`transformProcessNsPerRecord`、`transformRecordsIn`、`transformRecordsOut`、`subtaskCount`
|
+| Sink | `sinkBusyRatio` |
`sinkWriteNsPerRecord`、`sinkRecordsIn`、`sinkPrepareCommitNs`、`sinkCommitNs`、`sinkAbortNs`、`subtaskCount`
|
+
+节点主颜色应代表最符合该 vertex 类型的指标。例如 Source 使用读取或空闲比例，Transform 使用 transform busy
ratio，Sink 使用 sink busy ratio。
+
+### 边运行状态
+
+V1 只有带队列的 edge 才能暴露背压指标。每条图 edge 优先通过 REST 返回的 `targetVertexId` 合并最近一个
realtime edge point，必要时再从 `queueId` 解码。
+
+| 字段 | 含义 |
+|---|---|
+| `queueId` | realtime 聚合使用的稳定队列指标标识。 |
+| `targetVertexId` | 用来把队列指标映射回 DAG edge 的下游 vertex。 |
+| `bpRatio` | 当前 bucket 内生产端等待队列容量的时间占比。 |
+| `queueFillRatio` | 最近一次采样到的队列填充比例。 |
+| `queueSize` | 最近一次采样到的队列大小。 |
+| `queueCapacity` | 最近一次采样到的队列容量。 |
+
+edge 颜色应代表 `bpRatio`，edge 宽度应代表 `queueFillRatio`。详情抽屉应展示原始字段和最近 bucket 序列。
+
+## 稳定契约与 Best-Effort 信号
+
+运行时图需要区分稳定标识字段和有诊断价值但来自采样的信号。
+
+稳定契约字段：
+
+- `jobId`
+- `vertexId`
+- `queueId`
+- `targetVertexId`
+- `bucketMs`
+- `fromMs`
+- `toMs`
+- point 时间戳 `ts`
+- `subtaskCount`
+
+Best-effort 诊断字段：
+
+- busy ratio
+- idle ratio
+- 单条记录耗时估算
+- queue size
+- queue fill ratio
+- producer wait ratio
+- checkpoint 与错误摘要 badge
+
+Best-effort 字段在恢复、rescale、counter reset 或采样延迟附近可能波动。UI
应把它们表达为实时诊断信号，而不是审计口径的统计值。
+
+## 刷新、保留与成本
+
+V1 应保持现有刷新和保留模型：
+
+- worker counter 由 Active Master 收集
+- 收集结果只保留短窗口内存数据
+- REST 查询窗口默认 3 分钟
+- REST 查询窗口最大 10 分钟
+- UI 可以比 collector 更频繁刷新，但必须能接受连续刷新拿到相同 bucket
+- 默认不把 runtime graph 数据写入磁盘
+
+这样可以让运行时图的成本跟随现有 realtime observability，而不是新增一条轮询或持久化链路。
+
+## Checkpoint 与错误上下文
+
+Checkpoint 和错误信号应显示在图附近，但 V1 中仍保持独立契约：
+
+- checkpoint 概览与历史继续来自 checkpoint REST 接口
+- 作业异常与日志继续来自现有作业详情 API
+- 图上可以展示小型状态 badge 或入口链接，但 drill-down 应打开现有详情面板
+
+这可以避免在生命周期与保留规则尚未明确前，把 checkpoint 数据混入 realtime metrics endpoint。
+
+## 大 DAG 降级策略
+
+大 DAG 可能难以阅读，也会带来较高重绘成本。V1 应当降级，而不是强行在每个元素上渲染全部信号。
+
+推荐行为：
+
+- 保留拓扑渲染能力
+- 展示最热 vertices 与最阻塞 edges 的摘要健康表
+- DAG 较大时限制自动 fit 和动画
+- 保持选中节点或边后的 drill-down 能力
+- 不为了 UI 复杂度提高 master 采集频率
+
+具体实现 PR 应记录选择的大 DAG 阈值，并将其保持为 UI 渲染规则，而不是后端采样规则。
+
+## V1 交付计划
+
+1. 保持 Active Master realtime REST endpoint 作为 vertex 和 edge 运行状态来源。
+2. 根据匹配 `vertexId` 的最新 vertex point 渲染节点颜色。
+3. 根据匹配 `targetVertexId` 的最新 edge point 渲染 edge 颜色和宽度。
+4. 在现有详情抽屉中展示最近 point 序列和原始字段。
+5. 将 checkpoint 与错误信息作为邻近上下文入口，而不是向 realtime metrics 中嵌入新字段。
+6. 对过大的 DAG 提供降级视图，列出最热 vertices 与最阻塞 edges。
+7. 同步更新 REST、Web UI、运维文档的英文与中文说明。
+
+## 验收要求
+
+实现 PR 至少应覆盖这些检查：
+
+- 后端 realtime edge 和 vertex 响应测试覆盖当前字段与 `targetVertexId` 映射
+- UI 测试覆盖节点染色、edge 染色、edge 宽度和指标未开启时的行为
+- 大 DAG 降级使用确定性的合成 DAG 做测试
+- 文档明确 realtime window 仅为内存、best effort
+- V1 不引入新的持久化指标表、文件或写入路径
+
+## 相关文档
+
+- [实时可观测性](./realtime-observability.md)
+- [忙碌度与背压](./busyness-and-backpressure.md)
+- [Web UI](./web-ui.md)
+- [RESTful API V2](./rest-api-v2.md)
diff --git a/docs/zh/engines/zeta/web-ui.md b/docs/zh/engines/zeta/web-ui.md
index bebd335bcd..55598acd0f 100644
--- a/docs/zh/engines/zeta/web-ui.md
+++ b/docs/zh/engines/zeta/web-ui.md
@@ -78,6 +78,8 @@ Web UI 不负责提交作业，也不提供 cancel、stop、savepoint、restore
> 该能力需要作业侧开启 `env.engine.observability`（或满足默认开启条件），并按需配置
> `async_boundaries`、`split_sink_io` 等。
> 详细配置与指标说明请参考：[实时可观测性](realtime-observability.md)。
+运行时图的设计边界与大 DAG 降级规则请参考：[运行时执行图](runtime-execution-graph.md)。
+
### 已完成的作业
“已完成的作业”模块展示已进入终态的作业，例如 finished、failed、cancelled 或 savepoint
done。用户可以回看历史记录，并进入详情页查看配置、异常文本、引擎保留的指标和日志。
@@ -104,5 +106,6 @@ Web UI 不负责提交作业，也不提供 cancel、stop、savepoint、restore
- [REST API 与 Web UI](./rest-api-and-web-ui.md)
- [REST API V2](./rest-api-v2.md)
+- [运行时执行图](./runtime-execution-graph.md)
- [作业生命周期 API](./rest-api-job-lifecycle.md)
- [安全](./security.md)
