This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-12992-4aeb11368ad7957b079a1b5892603347820c717d in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 5d8937dd970ba12f2e29b9b17d4dc1ffcebb4e69 Author: Yohanes <[email protected]> AuthorDate: Tue Nov 17 15:52:59 2026 +0000 [Feature][AI CLI] Correlate realtime queue pressure with downstream work (#12992) --- seatunnel-cli/README.md | 46 +++- seatunnel-cli/README.zh-CN.md | 39 ++- seatunnel-cli/seatunnel_cli/cli.py | 15 +- .../seatunnel_cli/diagnostics/realtime.py | 218 +++++++++++++++ seatunnel-cli/tests/test_diagnose_cli.py | 112 ++++++++ seatunnel-cli/tests/test_diagnose_job.py | 293 +++++++++++++++++++++ 6 files changed, 717 insertions(+), 6 deletions(-) diff --git a/seatunnel-cli/README.md b/seatunnel-cli/README.md index 9067e83cfd..ba94d1109f 100644 --- a/seatunnel-cli/README.md +++ b/seatunnel-cli/README.md @@ -279,8 +279,8 @@ seatunnel "Read CSV files and write to Elasticsearch" --provider openai --model ### Diagnosing a Job `--diagnose` answers "why is this job not making progress?" from what the -engine already publishes. It is read-only -- a `GET /job-info/<id>` and a -`GET /jobs/checkpoints/<id>`, never a submit or a stop -- and uses no LLM at +engine already publishes. It is read-only -- GET requests for job state, +checkpoint history and optional realtime metrics, never a submit or a stop -- and uses no LLM at all, so it works with no API key and no provider configured: ```bash @@ -332,7 +332,47 @@ been slow since. If the cluster predates the checkpoint endpoint, or its monitor service is off, the rest of the diagnosis is printed unchanged, with one line noting that this part of the analysis did not run. -Every rule is deliberately limited to what one response can prove, because a +For running jobs, the CLI also reads +`/metrics/realtime/jobs/<id>/edges` and `/vertices` over the last three minutes. +With `env.engine.observability.enabled = true`, three consecutive completed +buckets can identify a queue that was persistently full and backpressured. +The output retains `queueId`, `targetVertexId`, the measured window, `bucketMs` and ratios. +Busy sink writes or transform processing in the same buckets make that +downstream vertex a **candidate to investigate**, rather than a proven root +cause. Sink prepare/commit time is checked separately, including when write +busyness is low; its sum is normalized by bucket duration and subtask count. + +Fixture-based output with five-second buckets (synthetic millisecond timestamps): + +```text +Job 852 (unnamed) — RUNNING + [warning] Queue -3 into vertex 1 was persistently backpressured in the matched window. Downstream sink +writes were busy too; investigate this vertex as a bottleneck candidate. (queueId=-3 targetVertexId=1 +buckets=3 window=0..15000ms bucketMs=5000 median bpRatio=0.80 queueFillRatio=0.90) +``` + +This points to a queue/vertex pair for investigation over a measured 15-second window; +it does not identify the cause. If observability is disabled, the same command reports a coverage gap: + +```text +Job 852 (unnamed) — RUNNING + [info] Realtime pressure checks unavailable: engine.observability.enabled is false. +``` + +The initial thresholds are backpressure ≥ 0.5 and fill ≥ 0.8 in all three +buckets; busy or prepare/commit time ≥ 0.5 supplies the corresponding work +evidence. Repeated points are not additional samples, the current partial +bucket is excluded, and endpoints must have matching bucket widths and an +overlapping window. The three buckets must be the latest complete buckets in +that shared engine window; older matches cannot replace missing or conflicting +recent data. Disabled, missing or insufficient data is reported as +unavailable while job/checkpoint diagnosis continues. Source read time alone +is not source attribution: synchronous downstream work can execute within +the source's `pollNext()`. +Backpressure ratios are preserved as reported by the engine, including values +above one; queue fill and vertex busy ratios are bounded by the engine. + +Every rule is deliberately limited to what its inputs can prove, because a wrong hint costs more than a missing one: - `restoreCount` counts restores since submission, so what it means depends on diff --git a/seatunnel-cli/README.zh-CN.md b/seatunnel-cli/README.zh-CN.md index ec7b0938f5..3929a84d4d 100644 --- a/seatunnel-cli/README.zh-CN.md +++ b/seatunnel-cli/README.zh-CN.md @@ -247,7 +247,7 @@ seatunnel "Read CSV files and write to Elasticsearch" --provider openai --model ### 诊断作业 `--diagnose` 用引擎已经暴露的信息回答"这个作业为什么不往前走"。它是只读的—— -只发 `GET /job-info/<id>` 和 `GET /jobs/checkpoints/<id>`,不会提交也不会停止 +只用 GET 读取作业状态、checkpoint 历史和可选的实时指标,不会提交也不会停止 作业——并且完全不调用 LLM,因此在没有 API key、没有配置任何提供商的情况下也能用: ```bash @@ -293,7 +293,42 @@ acknowledged it`),指向具体的某一环而不是整个作业。 退化、之后一直慢的作业同样有效。如果集群版本没有该 checkpoint 端点,或其监控服务 未开启,诊断的其余部分照常输出,并会额外打印一行说明这部分分析没有执行。 -每条规则都刻意只说单次响应能证明的事情,因为错误的提示比没有提示代价更大: +对于运行中的作业,CLI 还读取最近三分钟的 +`/metrics/realtime/jobs/<id>/edges` 和 `/vertices`。开启 +`env.engine.observability.enabled = true` 后,三个连续且已完成的时间桶 +可用于识别持续满载、持续背压的队列。输出保留 `queueId`、`targetVertexId`、 +观测时间窗口、`bucketMs` 桶宽和比率。如果相同时间桶里的 sink 写入或 transform 处理也很忙, +下游节点会被标记为**待排查的瓶颈候选**,而不是确定的根因。 +sink prepare/commit 耗时单独分析,即使写入忙碌度很低也可见;总耗时会按 +时间桶长度和子任务数归一化。 + +下面是五秒桶宽的测试样例输出,时间戳为合成的毫秒值: + +```text +Job 852 (unnamed) — RUNNING + [warning] Queue -3 into vertex 1 was persistently backpressured in the matched window. Downstream sink +writes were busy too; investigate this vertex as a bottleneck candidate. (queueId=-3 targetVertexId=1 +buckets=3 window=0..15000ms bucketMs=5000 median bpRatio=0.80 queueFillRatio=0.90) +``` + +它指出一个值得排查的队列和目标节点,观测窗口为 15 秒,并不判断根因。 +如果观测未开启,同一命令会说明缺少分析依据: + +```text +Job 852 (unnamed) — RUNNING + [info] Realtime pressure checks unavailable: engine.observability.enabled is false. +``` + +初始阈值为三个时间桶均满足背压 ≥ 0.5、队列填充 ≥ 0.8;忙碌度或提交耗时 +占比 ≥ 0.5 用于提供对应的执行证据。重复数据不算新样本,当前尚未完成的 +时间桶被排除,两次响应必须使用相同桶宽且窗口重叠。三个桶必须是引擎共同 +窗口中最近完成的桶;最近数据缺失或冲突时,不会用更早的匹配样本替代。 +观测关闭、数据缺失或样本不足会显示为不可分析,作业和 checkpoint 诊断仍继续。source read +耗时本身不能证明源端是瓶颈,因为同步下游操作也可能发生在 `pollNext()` 中。 +背压比率按引擎原始值显示,包括大于 1 的值;队列填充和节点忙碌比率则由 +引擎限制在 0 到 1 之间。 + +每条规则都刻意只说输入能证明的事情,因为错误的提示比没有提示代价更大: - `restoreCount` 统计的是自提交以来的重启次数,所以它的含义完全取决于 pipeline 当前的状态。`FINISHED` / `CANCELED` 的 pipeline 已经结束,不会输出任何内容; diff --git a/seatunnel-cli/seatunnel_cli/cli.py b/seatunnel-cli/seatunnel_cli/cli.py index 40fe715224..225fadfc93 100644 --- a/seatunnel-cli/seatunnel_cli/cli.py +++ b/seatunnel-cli/seatunnel_cli/cli.py @@ -1598,12 +1598,13 @@ def run_diagnose(job_id: str, console: Console) -> int: and no network beyond the cluster -- which is exactly the situation someone is in when a job is stuck in production and they want an answer now. - It is also read-only: two GETs, no submit, no stop, no savepoint. That is + It is also read-only: GETs only, no submit, no stop, no savepoint. That is what makes it safe to point at a live production job. """ from .connectors import _ENGINE_API_BASE from .diagnostics.checkpoints import diagnose_checkpoints from .diagnostics.jobs import as_int, diagnose_job, is_non_terminal, sort_findings + from .diagnostics.realtime import diagnose_realtime from .memory import redact_credentials if not _JOB_ID_RE.match(job_id): @@ -1738,6 +1739,18 @@ def run_diagnose(job_id: str, console: Console) -> int: findings + diagnose_checkpoints(checkpoints, now_ms=engine_now_ms) ) + # Completed jobs no longer have a realtime store. Optional metric failures + # must not discard job/checkpoint evidence or construct an LLM provider. + if status.upper() == "RUNNING": + try: + metric_base = f"{_ENGINE_API_BASE}/metrics/realtime/jobs/{job_id}" + edges = rest.request_json(f"{metric_base}/edges?windowMs=180000", timeout=10) + vertices = rest.request_json(f"{metric_base}/vertices?windowMs=180000", timeout=10) + except Exception as e: + edges, vertices = None, None + logger.debug("Realtime metrics for job %s unavailable: %s", job_id, e) + findings = sort_findings(findings + diagnose_realtime(edges, vertices)) + if not findings: # Only claim a clean bill of health when the rules actually had their # input; otherwise the notices above are the whole story. diff --git a/seatunnel-cli/seatunnel_cli/diagnostics/realtime.py b/seatunnel-cli/seatunnel_cli/diagnostics/realtime.py new file mode 100644 index 0000000000..28747a80e2 --- /dev/null +++ b/seatunnel-cli/seatunnel_cli/diagnostics/realtime.py @@ -0,0 +1,218 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""Correlate retained queue pressure with downstream work, without model I/O. + +These are candidates to investigate, not connector/root-cause attribution. +Only the latest three distinct, consecutive, completed and matched buckets +count; one spike, a partial bucket or three reads of one bucket do not. +""" + +from __future__ import annotations + +import math +from statistics import median + +from .jobs import Finding + +MIN_BUCKETS = 3 +BACKPRESSURE_RATIO = 0.5 +QUEUE_FILL_RATIO = 0.8 +BUSY_RATIO = 0.5 + + +def _integer(value): + if isinstance(value, bool) or not isinstance(value, (int, str)): + return None + try: + return int(value) + except ValueError: + return None + + +def _number(value): + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + return value if math.isfinite(value) and value >= 0 else None + + +def _ratio(value): + value = _number(value) + return value if value is not None and value <= 1 else None + + +def _window(data, key): + if not isinstance(data, dict) or not isinstance(data.get(key), list): + return None + values = tuple(_integer(data.get(k)) for k in ("fromMs", "toMs", "bucketMs")) + start, end, width = values + if any(v is None for v in values) or start < 0 or end <= start or width <= 0: + return None + return values + + +def _points(series, start, end, width): + """Index complete aligned buckets, discarding conflicting duplicate data.""" + indexed = {} + conflicts = set() + points = series.get("points") + if not isinstance(points, list): + return indexed + for point in points: + if not isinstance(point, dict): + continue + ts = _integer(point.get("ts")) + if ts is None or ts < start or ts + width > end or ts % width: + continue + if ts in indexed and indexed[ts] != point: + conflicts.add(ts) + indexed[ts] = point + return {ts: point for ts, point in indexed.items() if ts not in conflicts} + + +def _recent_run(timestamps, width, end): + """Require the window's latest complete buckets, not a stale matching run.""" + complete_end = (end // width) * width + required = [complete_end - i * width for i in range(MIN_BUCKETS, 0, -1)] + return required if all(ts in timestamps for ts in required) else [] + + +def _unavailable(reason): + return Finding( + "info", + "realtime_unavailable", + f"Realtime pressure checks unavailable: {reason}.", + ) + + +def diagnose_realtime(edges, vertices) -> list[Finding]: + """Read the engine's edges/vertices responses, using their shared clock window.""" + if any( + isinstance(data, dict) and data.get("enabled") is False + for data in (edges, vertices) + ): + return [_unavailable("engine.observability.enabled is false")] + edge_window, vertex_window = _window(edges, "edges"), _window(vertices, "vertices") + if edge_window is None or vertex_window is None: + return [_unavailable("an endpoint or its metric window is missing")] + if edge_window[2] != vertex_window[2]: + return [_unavailable("the responses have different bucket widths")] + start, end = max(edge_window[0], vertex_window[0]), min( + edge_window[1], vertex_window[1] + ) + width = edge_window[2] + if end - start < MIN_BUCKETS * width: + return [_unavailable("fewer than three complete buckets fit the shared window")] + + vertex_points = {} + for series in vertices["vertices"]: + if isinstance(series, dict): + vertex_id = _integer(series.get("vertexId")) + if vertex_id is not None: + vertex_points[vertex_id] = _points(series, start, end, width) + + findings = [] + matched_runs = 0 + incomplete_queues = 0 + for edge in edges["edges"]: + if not isinstance(edge, dict): + incomplete_queues += 1 + continue + queue_id, target = _integer(edge.get("queueId")), _integer( + edge.get("targetVertexId") + ) + if queue_id is None or target is None: + incomplete_queues += 1 + continue + ep = _points(edge, start, end, width) + vp = vertex_points.get(target, {}) + run = _recent_run(ep.keys() & vp.keys(), width, end) + if not run: + incomplete_queues += 1 + continue + # The engine clamps vertex/fill ratios, but publishes bpRatio without + # an upper clamp. Preserve values above one from delayed collection. + bp = [_number(ep[ts].get("bpRatio")) for ts in run] + fill = [_ratio(ep[ts].get("queueFillRatio")) for ts in run] + if any(v is None for v in bp + fill): + incomplete_queues += 1 + continue + matched_runs += 1 + if not all( + b >= BACKPRESSURE_RATIO and f >= QUEUE_FILL_RATIO for b, f in zip(bp, fill) + ): + continue + detail = f"Queue {queue_id} into vertex {target} was persistently backpressured in the matched window." + for key, label in ( + ("sinkBusyRatio", "sink writes"), + ("transformBusyRatio", "transform processing"), + ): + busy = [_ratio(vp[ts].get(key)) for ts in run] + if all(v is not None and v >= BUSY_RATIO for v in busy): + detail += f" Downstream {label} were busy too; investigate this vertex as a bottleneck candidate." + break + else: + detail += " The downstream work metrics do not establish the cause; low write busyness does not exclude commit pressure." + findings.append( + Finding( + "warning", + "realtime_queue_pressure", + detail, + f"queueId={queue_id} targetVertexId={target} buckets={len(run)} " + f"window={run[0]}..{run[-1] + width}ms bucketMs={width} " + f"median bpRatio={median(bp):.2f} queueFillRatio={median(fill):.2f}", + ) + ) + + for vertex_id, points in vertex_points.items(): + run = _recent_run(points, width, end) + if not run: + continue + ratios = [] + for ts in run: + point = points[ts] + prepare, commit = _number(point.get("sinkPrepareCommitNs")), _number( + point.get("sinkCommitNs") + ) + tasks = _integer(point.get("subtaskCount")) + if prepare is None or commit is None or tasks is None or tasks <= 0: + break + ratios.append((prepare + commit) / (width * 1_000_000 * tasks)) + if len(ratios) == len(run) and all(value >= BUSY_RATIO for value in ratios): + findings.append( + Finding( + "warning", + "realtime_sink_commit_pressure", + f"Vertex {vertex_id} spent substantial time preparing/committing across three completed buckets. " + "Investigate checkpoint/commit work; sinkBusyRatio measures writes only.", + f"vertexId={vertex_id} buckets={len(run)} window={run[0]}..{run[-1] + width}ms bucketMs={width} " + f"median prepare+commit ratio={median(ratios):.2f}", + ) + ) + if not matched_runs: + findings.append( + _unavailable( + "no three consecutive queue/target buckets were available for attribution" + ) + ) + elif incomplete_queues: + findings.append( + _unavailable( + f"{incomplete_queues} queue series lacked matched, usable buckets" + ) + ) + return findings diff --git a/seatunnel-cli/tests/test_diagnose_cli.py b/seatunnel-cli/tests/test_diagnose_cli.py index b878efd084..0d920ed4f1 100644 --- a/seatunnel-cli/tests/test_diagnose_cli.py +++ b/seatunnel-cli/tests/test_diagnose_cli.py @@ -31,6 +31,7 @@ import pytest from rich.console import Console from seatunnel_cli import cli as cli_module +from test_diagnose_job import _realtime_payloads class _Handler(BaseHTTPRequestHandler): @@ -645,3 +646,114 @@ def test_brackets_in_an_engine_message_cannot_corrupt_the_output(engine): flat = " ".join(out.split()) assert code == 0 assert "Table[/bold] row[nope] idx[0]" in flat + + +def _realtime_routes(edges, vertices, checkpoints=None, edge_status=200): + def script(handler): + if "/metrics/realtime/" in handler.path: + if "/edges?" in handler.path: + return edge_status, json.dumps(edges) + return 200, json.dumps(vertices) + if "/jobs/checkpoints/" in handler.path: + return 200, json.dumps(checkpoints or {"pipelines": []}) + return 200, json.dumps( + { + "jobId": "852", + "jobStatus": "RUNNING", + "diagnostics": { + "generatedAt": _NOW, + "stateTimestamps": {"RUNNING": _NOW - 5000}, + }, + } + ) + + return script + + +def test_realtime_pressure_reaches_model_free_read_only_cli(engine, monkeypatch): + edges, vertices = _realtime_payloads() + engine.script = _realtime_routes(edges, vertices) + + def unexpected_provider(*args, **kwargs): + raise AssertionError("diagnosis must not construct a provider") + + monkeypatch.setattr(cli_module.SeaTunnelCLI, "__init__", unexpected_provider) + + code, out = _run() + + assert code == 0 + assert "[warning] Queue -3 into vertex 1" in out + assert "sink writes were busy" in out + assert "queueId=-3 targetVertexId=1 buckets=3" in out + assert "window=0..15000ms" in out + assert "bucketMs=5000" in out + assert engine.paths == [ + "/job-info/852", + "/jobs/checkpoints/852", + "/metrics/realtime/jobs/852/edges?windowMs=180000", + "/metrics/realtime/jobs/852/vertices?windowMs=180000", + ] + + [email protected]("body", [{}, {"enabled": False}, None, []]) +def test_optional_realtime_data_gap_is_not_a_healthy_diagnosis(engine, body): + _, vertices = _realtime_payloads() + engine.script = _realtime_routes(body, vertices) + code, out = _run() + assert code == 0 + assert "Realtime pressure checks unavailable" in out + assert "Nothing to report" not in out + assert "persistently backpressured" not in out + + [email protected]("bp", [0.0, 0.8]) +def test_stale_realtime_samples_are_reported_as_a_gap_in_the_cli(engine, bp): + edges, vertices = _realtime_payloads(bp=bp) + edges["toMs"] = vertices["toMs"] = 22000 + for point in vertices["vertices"][0]["points"]: + point["sinkCommitNs"] = 4_000_000_000 + engine.script = _realtime_routes(edges, vertices) + + code, out = _run() + + assert code == 0 + assert "Realtime pressure checks unavailable" in out + assert "Nothing to report" not in out + assert "persistently backpressured" not in out + assert "substantial time preparing/committing" not in out + assert len(engine.paths) == 4 + + +def test_missing_realtime_endpoint_preserves_checkpoint_slowdown(engine): + _, vertices = _realtime_payloads() + engine.script = _realtime_routes( + {}, + vertices, + edge_status=404, + checkpoints={ + "pipelines": [{"pipelineId": 1, "history": _slow_checkpoint_history()}] + }, + ) + code, out = _run() + assert code == 0 + assert "getting slower" in out + assert "Realtime pressure checks unavailable" in out + assert len(engine.paths) == 3 + + +def test_malformed_vertex_json_preserves_other_diagnosis(engine): + edges, vertices = _realtime_payloads() + routed = _realtime_routes(edges, vertices) + engine.script = lambda h: (200, "not JSON") if "/vertices?" in h.path else routed(h) + code, out = _run() + assert code == 0 + assert "Realtime pressure checks unavailable" in out + assert "persistently backpressured" not in out + + +def test_a_pre_running_job_does_not_fetch_running_only_metrics(engine): + engine.script = _routed({"jobId": "852", "jobStatus": "PENDING"}, {"pipelines": []}) + code, out = _run() + assert code == 0 + assert engine.paths == ["/job-info/852", "/jobs/checkpoints/852"] + assert "Realtime pressure checks unavailable" not in out diff --git a/seatunnel-cli/tests/test_diagnose_job.py b/seatunnel-cli/tests/test_diagnose_job.py index 35b7f80b16..ca3846624d 100644 --- a/seatunnel-cli/tests/test_diagnose_job.py +++ b/seatunnel-cli/tests/test_diagnose_job.py @@ -32,6 +32,8 @@ from seatunnel_cli.diagnostics.jobs import ( STUCK_SECONDS, diagnose_job, ) +from seatunnel_cli.diagnostics.realtime import diagnose_realtime +import pytest NOW = 1_760_000_000_000 # fixed clock, so the stuck rule is not time-dependent @@ -732,3 +734,294 @@ def test_restore_count_of_true_is_not_counted_as_one(): ) assert findings == [] + + +def _realtime_payloads(bp=0.8, fill=0.9, busy=0.7, timestamps=(0, 5000, 10000)): + window = {"fromMs": 0, "toMs": 17000, "bucketMs": 5000} + edges = { + **window, + "edges": [ + { + "queueId": -3, + "targetVertexId": 1, + "points": [ + {"ts": ts, "bpRatio": bp, "queueFillRatio": fill} + for ts in timestamps + ], + } + ], + } + vertices = { + **window, + "vertices": [ + { + "vertexId": 1, + "points": [ + { + "ts": ts, + "sinkBusyRatio": busy, + "sourceReadRatio": 0.0, + "transformBusyRatio": 0.0, + "sinkPrepareCommitNs": 0, + "sinkCommitNs": 0, + "subtaskCount": 1, + } + for ts in timestamps + ], + } + ], + } + return edges, vertices + + +def test_realtime_matches_completed_buckets_and_keeps_queue_identity(): + edges, vertices = _realtime_payloads() + finding = _by_rule(diagnose_realtime(edges, vertices), "realtime_queue_pressure") + assert finding.severity == "warning" + assert "sink writes" in finding.detail + assert "candidate" in finding.detail + assert "queueId=-3 targetVertexId=1 buckets=3" in finding.evidence + assert "window=0..15000ms" in finding.evidence + assert "bucketMs=5000" in finding.evidence + assert "bpRatio=0.80 queueFillRatio=0.90" in finding.evidence + + [email protected]("width", [1000, 7000]) +def test_realtime_evidence_reports_actual_bucket_width(width): + edges, vertices = _realtime_payloads(timestamps=(0, width, 2 * width)) + for data in (edges, vertices): + data["bucketMs"] = width + data["toMs"] = 3 * width + for point in vertices["vertices"][0]["points"]: + point["sinkCommitNs"] = int(width * 1_000_000 * 0.8) + + findings = diagnose_realtime(edges, vertices) + for rule in ("realtime_queue_pressure", "realtime_sink_commit_pressure"): + evidence = _by_rule(findings, rule).evidence + assert f"bucketMs={width}" in evidence + assert f"window=0..{3 * width}ms" in evidence + + +def test_realtime_transient_spike_is_not_sustained_pressure(): + edges, vertices = _realtime_payloads(bp=0.1) + edges["edges"][0]["points"][-1]["bpRatio"] = 0.9 + assert diagnose_realtime(edges, vertices) == [] + + +def test_realtime_backpressure_with_an_empty_queue_is_not_attributed(): + edges, vertices = _realtime_payloads(fill=0.1) + assert diagnose_realtime(edges, vertices) == [] + + +def test_realtime_multiple_queues_into_one_vertex_are_not_collapsed(): + import copy + + edges, vertices = _realtime_payloads() + second = copy.deepcopy(edges["edges"][0]) + second["queueId"] = -2 + edges["edges"].append(second) + findings = diagnose_realtime(edges, vertices) + assert len(findings) == 2 + assert "queueId=-3" in findings[0].evidence + assert "queueId=-2" in findings[1].evidence + + +def test_realtime_orders_points_by_timestamp_not_array_order(): + edges, vertices = _realtime_payloads() + expected = diagnose_realtime(edges, vertices) + edges["edges"][0]["points"].reverse() + vertices["vertices"][0]["points"].reverse() + assert diagnose_realtime(edges, vertices) == expected + + +def test_realtime_uses_latest_buckets_not_an_old_pressure_episode(): + edges, vertices = _realtime_payloads( + timestamps=(0, 5000, 10000, 15000, 20000, 25000) + ) + edges["toMs"] = vertices["toMs"] = 31000 + for point in edges["edges"][0]["points"][3:]: + point["bpRatio"] = 0 + assert diagnose_realtime(edges, vertices) == [] + + [email protected]("timestamps", [(0, 0, 0), (0, 10000, 15000), (0, 5000, 15000)]) +def test_realtime_repeated_partial_and_missing_buckets_are_not_extra_evidence( + timestamps, +): + edges, vertices = _realtime_payloads(timestamps=timestamps) + findings = diagnose_realtime(edges, vertices) + assert _rules(findings) == ["realtime_unavailable"] + + +def test_realtime_conflicting_duplicate_points_are_discarded(): + edges, vertices = _realtime_payloads() + edges["edges"][0]["points"].append({"ts": 0, "bpRatio": 0, "queueFillRatio": 0}) + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + + [email protected]("endpoint", ["edges", "vertices"]) [email protected]("gap", ["missing", "conflicting"]) +def test_realtime_recent_gap_cannot_fall_back_to_an_old_pressure_episode(endpoint, gap): + edges, vertices = _realtime_payloads(timestamps=(0, 5000, 10000, 15000)) + edges["toMs"] = vertices["toMs"] = 22000 + points = (edges if endpoint == "edges" else vertices)[endpoint][0]["points"] + if gap == "missing": + points.pop() + else: + conflicting = {**points[-1]} + conflicting["bpRatio" if endpoint == "edges" else "sinkBusyRatio"] = 0 + points.append(conflicting) + + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + + +def test_realtime_stale_healthy_samples_are_a_coverage_gap(): + edges, vertices = _realtime_payloads(bp=0, fill=0, busy=0) + edges["toMs"] = vertices["toMs"] = 27000 + + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + + +def test_realtime_old_commit_pressure_is_not_current_evidence(): + edges, vertices = _realtime_payloads(bp=0, fill=0, busy=0) + edges["toMs"] = vertices["toMs"] = 22000 + for point in vertices["vertices"][0]["points"]: + point["sinkCommitNs"] = 4_000_000_000 + + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + + +def test_realtime_window_ending_at_a_boundary_includes_the_just_completed_bucket(): + edges, vertices = _realtime_payloads() + edges["toMs"] = vertices["toMs"] = 15000 + + finding = _by_rule(diagnose_realtime(edges, vertices), "realtime_queue_pressure") + assert "window=0..15000ms" in finding.evidence + + +def test_realtime_freshness_uses_the_shared_engine_window(): + edges, vertices = _realtime_payloads(timestamps=(0, 5000, 10000, 15000)) + edges["toMs"] = 22000 + vertices["toMs"] = 17000 + + finding = _by_rule(diagnose_realtime(edges, vertices), "realtime_queue_pressure") + assert "window=0..15000ms" in finding.evidence + + +def test_realtime_requires_matching_target_and_timestamps(): + edges, vertices = _realtime_payloads() + vertices["vertices"][0]["vertexId"] = 999 + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + vertices["vertices"][0]["vertexId"] = 1 + for point in vertices["vertices"][0]["points"]: + point["ts"] += 5000 + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + + [email protected]("bad", [None, [], {}, {"enabled": False}]) +def test_realtime_missing_or_disabled_is_not_healthy(bad): + _, vertices = _realtime_payloads() + assert _rules(diagnose_realtime(bad, vertices)) == ["realtime_unavailable"] + + +def test_realtime_empty_enabled_store_still_has_no_evidence(): + edges, vertices = _realtime_payloads() + edges["enabled"] = vertices["enabled"] = True + edges["edges"] = vertices["vertices"] = [] + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + + +def test_realtime_window_and_bucket_widths_must_agree(): + edges, vertices = _realtime_payloads() + vertices["bucketMs"] = 1000 + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + vertices["bucketMs"] = 5000 + vertices["fromMs"] = 20000 + vertices["toMs"] = 40000 + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + + [email protected]( + "bad", [None, True, float("nan"), float("inf"), -0.1, 1.2, "0.9"] +) +def test_realtime_invalid_ratios_are_not_pressure(bad): + edges, vertices = _realtime_payloads() + edges["edges"][0]["points"][0]["queueFillRatio"] = bad + assert _rules(diagnose_realtime(edges, vertices)) == ["realtime_unavailable"] + + +def test_realtime_source_read_time_does_not_blame_the_source(): + edges, vertices = _realtime_payloads(busy=0) + for point in vertices["vertices"][0]["points"]: + point["sourceReadRatio"] = 0.99 + finding = _by_rule(diagnose_realtime(edges, vertices), "realtime_queue_pressure") + assert "do not establish the cause" in finding.detail + assert "source bottleneck" not in finding.detail + assert "sink writes were busy" not in finding.detail + + +def test_realtime_transform_busy_time_correlates_with_the_pressure(): + edges, vertices = _realtime_payloads(busy=0) + for point in vertices["vertices"][0]["points"]: + point["transformBusyRatio"] = 0.8 + finding = _by_rule(diagnose_realtime(edges, vertices), "realtime_queue_pressure") + assert "transform processing" in finding.detail + + +def test_realtime_commit_pressure_is_visible_even_when_write_busyness_is_zero(): + edges, vertices = _realtime_payloads(bp=0, fill=0, busy=0) + for point in vertices["vertices"][0]["points"]: + point.update( + subtaskCount=2, + sinkPrepareCommitNs=1_000_000_000, + sinkCommitNs=5_000_000_000, + ) + findings = diagnose_realtime(edges, vertices) + assert _rules(findings) == ["realtime_sink_commit_pressure"] + assert "writes only" in findings[0].detail + assert "ratio=0.60" in findings[0].evidence + + +def test_realtime_commit_time_is_normalized_by_bucket_and_subtask_count(): + edges, vertices = _realtime_payloads(bp=0, fill=0, busy=0) + for point in vertices["vertices"][0]["points"]: + point.update(subtaskCount=2, sinkCommitNs=3_000_000_000) + assert diagnose_realtime(edges, vertices) == [] + + +def test_realtime_one_slow_commit_is_not_persistent_pressure(): + edges, vertices = _realtime_payloads(bp=0, fill=0, busy=0) + vertices["vertices"][0]["points"][-1]["sinkCommitNs"] = 5_000_000_000 + assert diagnose_realtime(edges, vertices) == [] + + +def test_realtime_one_healthy_queue_does_not_hide_another_queues_data_gap(): + edges, vertices = _realtime_payloads(bp=0, fill=0) + edges["edges"].append({"queueId": -2, "targetVertexId": 1, "points": []}) + finding = _by_rule(diagnose_realtime(edges, vertices), "realtime_unavailable") + assert "1 queue series lacked matched" in finding.detail + + +def test_realtime_large_vertex_ids_keep_integer_precision(): + edges, vertices = _realtime_payloads() + target = 852362670771666945 + edges["edges"][0]["targetVertexId"] = str(target) + vertices["vertices"][0]["vertexId"] = target + finding = _by_rule(diagnose_realtime(edges, vertices), "realtime_queue_pressure") + assert f"targetVertexId={target}" in finding.evidence + + +def test_realtime_current_partial_bucket_cannot_complete_a_pressure_run(): + edges, vertices = _realtime_payloads(timestamps=(0, 5000, 10000, 15000)) + edges["edges"][0]["points"][0]["bpRatio"] = 0 + # 15000..20000 is not complete at toMs=17000. Only the current partial + # point would make the last three buckets look persistently pressured. + assert diagnose_realtime(edges, vertices) == [] + + +def test_realtime_preserves_engine_backpressure_ratios_above_one(): + # RealtimeMetricsService.edgePoint does not clamp bpRatio, unlike its + # vertex/fill ratios. A delayed collection can report more than one. + edges, vertices = _realtime_payloads(bp=1.2) + finding = _by_rule(diagnose_realtime(edges, vertices), "realtime_queue_pressure") + assert "bpRatio=1.20" in finding.evidence
