Hello! I wanted to bring up something I noticed about the current Ollama package in Fedora (0.12.11 / 0.24.0 in rawhide). The package ships with ROCm and Vulkan GPU backends, but there is no CUDA backend at all. This means users with NVIDIA GPUs can only use the Vulkan path for GPU acceleration, which is noticeably slower than CUDA for LLM inference workloads.
I'm running a laptop with an RTX 3060 (6 GB VRAM) and 64 GB of RAM. Many of the models I run don't fit entirely in VRAM, so some layers get offloaded to the CPU. In this scenario, both the GPU and CPU inference paths matter a lot. On the CPU side, the package already does a great job by shipping multiple architecture-specific builds (Alder Lake, Haswell, Ice Lake, etc.) with the right SIMD flags for each, so the layers that run in RAM are well optimized. But on the GPU side, the layers that should be running fast in VRAM are going through Vulkan instead of CUDA. The result is that half of the inference pipeline is properly optimized and the other half is not, which becomes especially noticeable with partially offloaded models where both paths are active at the same time. I know CUDA is proprietary and cannot live in Fedora's official repositories, which is why the package only ships ROCm and Vulkan today. But the package already splits GPU backends into separate subpackages (ollama-rocm, ollama-vulkan), so the structure for an ollama-cudasubpackage is already in place. Since the NVIDIA drivers and CUDA toolkit are already available through RPM Fusion, it seems like a natural fit to offer ollama-cuda there as well, following the same pattern. This would save NVIDIA users from having to compile from source or fall back to the upstream install script (which bypasses the package manager entirely and breaks on every kernel or driver update, as discussed in the Getting ollama to use GPU thread (Fedora Discussion) <https://discussion.fedoraproject.org/t/getting-ollama-to-use-gpu/181499/1>. It is worth noting that a community COPR already solves this, the fachep/ollama repository <https://copr.fedorainfracloud.org/coprs/fachep/ollama> packages an ollama-cuda subpackage built against CUDA 12.8, covering a wide range of GPU architectures from Maxwell to Blackwell. Given that this work already exists and is functional, would it be possible to incorporate it into the official Fedora package as an RPM Fusion dependency, or coordinate with the COPR maintainer to move it into RPM Fusion proper? Thanks for all the work on the Ollama package. The CPU variant builds and the overall packaging quality are really solid. Priscila.
-- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
