On 2026-07-31 09:40, Dr. Arne Babenhauserheide wrote:
Jean Louis <[email protected]> writes:
Majority of models you named as data assets are not free. Though some
are.
Do you know an easy to use option (and tutorial …) to run a fully free
LLM locally and use it from Emacs?
Oh, yes doctor, I am glad you ask.
Yes, so I recommend llama.cpp: LLM inference in C/C++:
https://github.com/ggml-org/llama.cpp to start with.
Prereqs (Debian): libvulkan-dev, glslc, spirv-headers — sudo apt install
glslc spirv-headers
And I have small .bash_functions function to recommend:
build_llama.cpp_vulkan() {
(
set -x;
pushd /mnt/nvme0n1/LLM/git/llama.cpp; # adopt your own directory
here
git pull && git log;
rm -rf build;
cmake -B build -DGGML_VULKAN=ON -DGGML_BLAS=ON
-DGGML_BLAS_VENDOR=OpenBLAS -DCMAKE_BUILD_TYPE=Release;
watch_cpu_info;
cmake --build build --config Release -j$(nproc);
cd build;
sudo make install -j$(nproc);
sudo /sbin/ldconfig;
popd;
)
}
Run: llama-cli -m model.gguf -ngl 99
Verify: llama-cli --list-devices shows Vulkan0 only.
I am not sure how much is that really free, and if it uses nvidia
drivers or not.
Vulkan runs on the NVIDIA GPU through NVIDIA's own Vulkan driver (the
ICD that ships with the nvidia driver package). So:
- NVIDIA kernel driver: yes, still needed
- NVIDIA Vulkan userspace driver (icd): yes
- CUDA toolkit, nvcc, cuBLAS, CUDA runtime, CUDA kernels: no — none of
it is compiled in or loaded anymore
To avoid proprietary NVIDIA software, specifically the CUDA ecosystem,
the most relevant Intel GPU to consider is the Intel Arc Pro B60. I
would say 2 of them. Flagship AI inference card from Intel; supported in
open-source AI software stacks (like llama.cpp).
Yet many models can run efficiently without GPU, so you could try those
of 4B on hugginface.co by downloading appropriate gguf files.
Example with Apertus:
llama-server -m
/mnt/data/LLM/quantized/Apertus-8B-Instruct-2509-Q6_K.gguf --device
Vulkan0
+ llama-server -m
/mnt/data/LLM/quantized/Apertus-8B-Instruct-2509-Q6_K.gguf --device
Vulkan0
0.00.115.756 I cmn common_param: common_params_print_info: verbosity =
3 (adjust with the `-lv N` CLI arg)
0.00.116.295 W srv llama_server: -----------------
0.00.116.298 W srv llama_server: CORS is set to allow all origins ('*')
and no API key is set
0.00.116.298 W srv llama_server: this can be a security risk
(cross-origin attacks)
0.00.116.299 W srv llama_server: more info:
https://github.com/ggml-org/llama.cpp/pull/25655
0.00.116.299 W srv llama_server: -----------------
0.00.117.520 I srv load_model: loading model
'/mnt/data/LLM/quantized/Apertus-8B-Instruct-2509-Q6_K.gguf'
0.00.565.360 W load: special_eos_id is not in special_eog_ids - the
tokenizer config may be incorrect
0.04.577.347 I srv load_model: initializing, n_slots = 4, n_ctx_slot
= 65536, kv_unified = 'true'
0.04.599.728 I srv llama_server: model loaded
0.04.599.736 I srv llama_server: listening on http://127.0.0.1:8080
and then you run some of already available LLM packages from within
Emacs to access it. You can change server, host, port. I have my own
functions for that, and you can make your own as well.
I am willing to pay it if someone is interested to send me the card to
Uganda. It is not easy ordering from here.
--
Jean Louis
---
via emacs-tangents mailing list
(https://lists.gnu.org/mailman/listinfo/emacs-tangents)
