weiqingy commented on issue #893: URL: https://github.com/apache/flink-agents/issues/893#issuecomment-4939647474
Root-cause update: the segfault is a per-runner-VM lottery, not something we can pin or configure away. I tried hard to reproduce and stabilize it, and the evidence converges on a hardware/environment cause tied to the specific GitHub-hosted runner VM: - **Not reproducible in isolation.** A stress harness on the same `ubuntu-latest` image ran thousands of concurrent, grammar-constrained (`format=json`), model-churning qwen3 inferences — 0 crashes. - **Not reproducible even with the real workload.** Looping the actual segfaulting Java integration tests (`YamlLoaderIntegrationTest`, `ChatModelIntegrationTest`, `ReActAgentTest`) ~90 times across 6 runner VMs — 0 crashes — while CI on this repo crashes ~18–20% per job in the same window. - **Constraining Ollama did not change anything.** An A/B of `OLLAMA_NUM_PARALLEL=1` + `OLLAMA_MAX_LOADED_MODELS=1` vs default came back 0 vs 0 — inconclusive, because the baseline would not crash either. - **Not a version change.** Ollama v0.31.2 has been the installed "latest" since 07-06, spanning both the green (07-07/08) and red (07-10) days, so a version pin cannot explain or fix the onset. Putting this together with the earlier signals — the ~19-minute in-job wedge (once a job's VM starts crashing it stays crashing), the failures roaming between matrix jobs, and the version-neutral pass/fail within a single run — the model that fits everything is: a subset (~20%) of GitHub's runner VMs are "cursed" by a hardware/microarchitecture condition that trips the underlying llama.cpp segfault, and a given VM is effectively deterministic (good or bad). The dated onset (~07-09) lines up with a GitHub runner-image rollout that made that subset bad, not with anything in this repo or in Ollama. Consequence: nothing that stays on the *same* VM can fix it — not a bigger in-suite retry (retries into the same crashed server), not an in-process Ollama restart, not a version pin, not an Ollama resource flag. The only reliable escape is to re-run the failed job on a **fresh runner**, which re-rolls the VM (~80% land healthy). Mitigation opened as #895: a `workflow_run` companion that, when CI fails, re-runs only the failed integration-test legs on a fresh runner (unit-test/lint/build failures are never re-run, so real regressions stay red). It cannot demonstrate itself on its own PR — `workflow_run` companions run only from the default branch — but once merged it applies to every subsequent CI run: a flaky integration-test leg auto-heals to green while a genuine failure that reproduces on every fresh runner stays red. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
