weiqingy commented on issue #893: URL: https://github.com/apache/flink-agents/issues/893#issuecomment-4937793102
Additional evidence from re-running the two CI-hardening PRs (#891, #892): the failures **roam between matrix jobs from one run to the next**, while the root cause stays the same. This is the clearest signature that it is the intermittent `llama-server` crash rather than any deterministic bug — a real regression would fail the *same* job every time. Same two PRs, two consecutive CI runs each: - Run 1 — #891 failed `it-python [3.11] [flink-1.20]` + `it-python [3.12] [flink-2.1]`; #892 failed `it-java [java-21] [flink-2.0]` + `it-python [3.11] [flink-1.20]`. - Run 2 (fresh trigger) — #891 failed `cross-language` + `it-java [flink-1.20]` + `it-java [java-21] [flink-2.0]`; #892 failed `it-java [flink-1.20]` + `it-python [3.11] [flink-1.20]`. Spot-checking the failing jobs confirms the same signatures: the roaming ones (`it-java`, `it-python`, `cross-language` across various Flink versions) all show the `llama-server process has terminated: signal: segmentation fault` crash (failure A), and `it-python [flink-1.20]` shows the py4j gateway death (failure B). Both survive the existing `--reruns 2` / `rerunFailingTestsCount=2`. Rough flake rate: across the Ollama-dependent jobs (~11 per run), 2–3 crash each run, i.e. an order of ~20% per-job segfault rate. That puts the probability of an all-green run at roughly `0.8^11 ≈ 8%` — so re-triggering (even via a fresh commit) is not a reliable way to get green; it just relocates which job crashes, and it burns shared runner capacity doing so. This reinforces the direction in the description: raising the retry count cannot help (the retry fires ~5s later into the same dead `ollama serve`), so the mitigation has to restart Ollama on the "process has terminated" signature before retrying — and that restart needs verifying against the runner before it is treated as a fix, since the crash may simply recur on the fresh server. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
