weiqingy opened a new issue, #893: URL: https://github.com/apache/flink-agents/issues/893
### Search before asking - [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar. ### Description Two integration-test CI jobs fail intermittently for reasons unrelated to the code under test, and — importantly — these failures **survive the test reruns already configured for the live-LLM e2e suite**. They cost every contributor a "is this red mine?" investigation on unrelated PRs. The existing mitigation from #716/#717 retries flaky tests: Python e2e runs with `pytest --reruns 2 --reruns-delay 5` and Java e2e with `-Dsurefire.rerunFailingTestsCount=2`. Recent failing runs show the reruns firing (`16 rerun`, `56 rerun` in the pytest summaries) and the jobs still ending red. So turning the retry count up will not help — a smarter mitigation is needed. **Failure A — local `llama-server` segfault (the dominant one).** The Ollama model server crashes at inference time while serving `qwen3:1.7b`: ``` ollama._types.ResponseError: llama-server process has terminated: signal: segmentation fault (core dumped) io.github.ollama4j.exceptions.OllamaException: llama-server process has terminated ``` It is not a version regression: within a single CI run every matrix job installs the same-day Ollama binary, yet one job crashes while its siblings pass (e.g. `it-java [flink-2.0]` crashed while `it-java` on 1.20/2.1/2.2/2.3 passed in the same run). Because it varies at a fixed binary, pinning an Ollama version would add reproducibility but would not stop the crash. The model pulls fine (1.4 GB), so it is an inference-time crash, not a download issue. Why the current reruns miss it: a rerun retries the same test ~5s later against the **same** `ollama serve` process whose `llama-server` subprocess just died, so it hits the dead server again. A rerun only helps if Ollama (or the model) is restarted between attempts. **Failure B — PyFlink gateway death on Flink 1.20 remote-submit e2e.** Only on `it-python [python-3.11] [flink-1.20]`; when it fails, all six PyFlink remote-submit e2e tests fail together on a dead py4j gateway: ``` py4j.protocol.Py4JError: org does not exist in the JVM ``` (The odd `KeyError: 'p'` seen alongside it is raised inside `py4j/protocol.py` when the client reads a garbled type-code byte off a broken gateway channel — a symptom of the same dead gateway, not repo code.) This is pre-existing and intermittent (~1 in 10 recent `main` runs) and also survives the reruns, which suggests the minicluster/gateway stays broken for the whole retry window. ### Proposed direction (to investigate, not prescribe) - **A:** detect the "process has terminated" signature and **restart `ollama serve` (or re-pull the model) before retrying**, rather than retrying into the dead server — e.g. a rerun condition hook, or a shell retry loop around the e2e step in `tools/ut.sh`. Verify a restart actually clears the crash in the runner before committing to it. - **B:** determine whether a fresh minicluster per retry attempt recovers the gateway; if the flake is purely environmental, a tracking note may be the honest outcome rather than a code change. - Any mitigation must stay **scoped to the live-LLM / external-service e2e tests** so it does not mask genuine intermittent regressions in unit tests. This extends the flaky-CI line started in #716/#717/#719. It is independent of #889 (which hardens the Ollama install and bounds job runtime). ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
