weiqingy opened a new pull request, #904: URL: https://github.com/apache/flink-agents/pull/904
Linked issue: #893 ### Purpose of change `llama-server` ships one `libggml-cpu-<microarch>` build per instruction set and loads the best match for the host CPU at load time. The hosted runner pool is mixed, so the kernels that serve a request depend on which machine a job happens to land on, and jobs that land on the newer hosts intermittently die with `llama-server process has terminated: signal: segmentation fault`. Upstream reports the same crash on hosts with the newer vector extensions (ollama/ollama#17006, open). That explains the shape of the flake described on #893: it is deterministic per runner rather than random, it survives the in-suite reruns (they retry into the same wedged process), and it never reproduced in isolation, because a fork probe kept landing on unaffected hardware. A census of 40 `ubuntu-latest` runners shows how mixed the pool is: | Runner CPU | Count | AVX-512 | AMX | |---|---|---|---| | AMD EPYC 7763 | 16 | no | no | | AMD EPYC 9V74 | 11 | no | no | | AMD EPYC 9V74 | 5 | yes (incl. BF16) | no | | Intel Xeon Platinum 8573C | 3 | yes | no | | Intel Xeon Platinum 8370C | 3 | yes | no | | Intel Xeon 6973P-C | 2 | yes (incl. BF16) | yes | 13 of 40 runners expose AVX-512, and `llama-server` selects `skylakex` / `icelake` / `zen4` / `sapphirerapids` on those. The remaining runners fall back to the AVX2 build, which does not crash. The crash rate measured on `main` was 6 segfaults in 44 integration jobs (~14%), which is the same order as the share of the pool running those builds. This removes the AVX-512-bearing builds after install, so `llama-server` falls back to the AVX2 build that every runner in the pool can execute. It also logs the runner CPU and the remaining builds, so a future failure can be attributed to the microarchitecture that served it. The exact extension at fault is not pinned down: AMX, AVX-512 and AVX512-BF16 are each present on a share of the pool close enough to the observed crash rate that the data here cannot separate them. The change therefore removes all of them rather than guessing, and falls back to the one build the whole pool shares. Note for reviewers: `OLLAMA_LLM_LIBRARY` is not a usable lever for this on 0.31.x. It is consumed in `discover/runner.go` inside `GPUDevices()`, where it filters GPU library directories by name (`cuda_v12`, `rocm_v6`, `mlx_*`); a CPU value matches no directory and has no effect on which CPU build `llama-server` loads. ### Tests Validated on a fork, since the crash only reproduces on the hosted runner pool. Three full CI runs with this change: **33 integration jobs, 0 segfaults.** At the ~14% per-job rate measured on `main`, ~4-5 segfaults were expected, and the chance of 33 clean jobs by luck is under 1%. The suspect hardware did come up during those runs (3 jobs on AVX512-BF16 hosts, 1 on an AMX host) and none of them crashed. The one unrelated failure in those 33 jobs was `EmbeddingIntegrationTest` on an AVX2-only host, with no segfault in the job; that host has no AVX-512 build to remove, so it ran the same kernels either way. No measurable runtime cost: median `it-python` 16m and `it-java` 10m, unchanged from the same jobs without the change. ### API No public API changes; this only affects CI. ### Documentation - [ ] `doc-needed` - [x] `doc-not-needed` - [ ] `doc-included` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
