weiqingy commented on issue #893: URL: https://github.com/apache/flink-agents/issues/893#issuecomment-4955148311
**Update — one of the two failure modes has been root-caused and fixed, which narrows this issue.** The earlier triage here split the flaky IT failures into **A** (the `llama-server` segfault) and **B** (the PyFlink py4j gateway death — `org does not exist in the JVM` / `Could not found the Java class 'org.apache.flink.configuration.Configuration'`), and recorded B as environmental, `flink-1.20`-specific, and out of scope. That no longer holds: B was an in-repo test bug, and #896 has fixed it. The MCP e2e test started its server with `multiprocessing.Process`, forking the already multi-threaded PyFlink + gRPC test process and corrupting the shared Py4J gateway, which then killed every PyFlink test that ran after it. Corrections to the record, from [run 29222634767](https://github.com/apache/flink-agents/actions/runs/29222634767) and [run 29221169818](https://github.com/apache/flink-agents/actions/runs/29221169818): - **Not environmental** — in-repo test code, and fixable. - **Not `flink-1.20`-only** — it appears on `flink-2.1` and `flink-2.2` (java-17 / python-3.12). - **Not a missing-dist-JAR classpath problem** — the JARs were present; the *gateway* was corrupt, which is also what explains the intermittency. - **Not a minor mode** — across the last two red `main` runs, 3 of the 4 failed jobs carried the py4j signature and only 1 was the segfault. The failure boundary in those logs is the MCP test itself, with the cascade starting on the very next test — the same boundary #896 uses in its own verification: ``` MCP server mode: without_prompts MCP server endpoint: http://127.0.0.1:8001/mcp FAILED execute_test.py::test_durable_execute_basic_flink FAILED execute_test.py::test_durable_execute_multiple_calls_flink FAILED ... (and every remaining PyFlink e2e test) ``` The first post-#896 `main` run ([29227640397](https://github.com/apache/flink-agents/actions/runs/29227640397)) confirms it: **10 of 11 IT jobs green, with no py4j signature anywhere in the run**, including the `flink-2.1` and `flink-2.2` legs that had been failing on it. The one failed job, `it-python [python-3.11] [flink-2.0]`, is pure failure A — 16 hits of `llama-server process has terminated: signal: segmentation fault`, and nothing else. **So the scope of this issue narrows to A alone.** A is unchanged: a per-runner-VM effect, not pinnable to an ollama version, with no honest in-code fix. The failure-rate figures quoted earlier in this thread counted B and are superseded — the "~8% chance of an all-green IT matrix" number in particular should not be relied on. **For #895** (auto-rerun failed IT jobs on a fresh runner), the run above is a precise statement of what remains: a single segfaulted job still turns an otherwise-green 11-job matrix red, and a fresh runner is still the only thing that escapes A. That is exactly the case #895 covers — rerun the one failed job rather than re-rolling all eleven. It's worth watching a few more `main` runs to see how often A fires now that B is gone; if it proves rare, closing #895 is the reasonable outcome rather than carrying a workflow for a flake that no longer bites. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
