This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch tui-ollama-tab in repository https://gitbox.apache.org/repos/asf/camel.git
commit c5259a0393def1d69efe5b1051868f2bfd73b989 Author: Claus Ibsen <[email protected]> AuthorDate: Thu Sep 17 13:05:08 2026 +0200 CAMEL-24794: camel-jbang - TUI manual explains working with a local Ollama model New subsection under the AI provider docs: how a question is spent with Ollama (load, prefill, decode and time to first token, with figures measured on an M4 Pro with qwen3.6:35b-a3b), why one question costs several requests and how the prompt cache keeps that affordable, the 32k context window, compaction and its re-prefill cost, the rule to keep every Ollama client on the same context size, how the model is chosen, where to look (Ollama tab, /context, /usage, Run Doctor), remote and containerised Ollama, and links to the local model benchmark and GenAI observability blog posts. The Ollama tab section points to it. Co-Authored-By: Claude Fable 5.1 <[email protected]> Signed-off-by: Claus Ibsen <[email protected]> --- .../modules/ROOT/pages/camel-jbang-tui.adoc | 69 +++++++++++++++++++++- 1 file changed, 68 insertions(+), 1 deletion(-) diff --git a/docs/user-manual/modules/ROOT/pages/camel-jbang-tui.adoc b/docs/user-manual/modules/ROOT/pages/camel-jbang-tui.adoc index 2d3f8a0373a6..64db50b05aa8 100644 --- a/docs/user-manual/modules/ROOT/pages/camel-jbang-tui.adoc +++ b/docs/user-manual/modules/ROOT/pages/camel-jbang-tui.adoc @@ -451,7 +451,8 @@ second. Against a remote or containerised Ollama the tab keeps the model, per-re and says which panels are unavailable. Press *r* to reset the request log and the session totals, *F5* to refresh immediately. The same data -is available to AI agents through the `tui_get_ollama` MCP tool. +is available to AI agents through the `tui_get_ollama` MCP tool. For what the figures mean for the AI +panel and which knobs to turn, see <<_working_with_a_local_ollama_model>>. == Process Information @@ -904,6 +905,72 @@ all tools to a local model too, `/tools core` to trim the set for a hosted one, model loaded for 30 minutes and use a 32k context window (`OLLAMA_CONTEXT_LENGTH` overrides it), so follow-up questions reuse the cached prompt instead of reloading the model. +==== Working with a local Ollama model + +A local model is not a slower version of a hosted one; it spends its time differently, and the TUI +shows you where. This section explains what a question costs with Ollama and which knobs matter, +with figures measured on an Apple M4 Pro (64 GB) running `qwen3.6:35b-a3b`. + +*How a question is spent.* Ollama answers in three phases, and every timing the TUI shows maps to one +of them: + +1. *Load*: if the model is not in memory (first question, the keep-alive expired, or a request asked for + a different context size) Ollama starts a runner and loads the weights: 10 to 20 seconds for a 22 GB + model. The TUI asks Ollama to keep the model loaded for 30 minutes after each request. +2. *Prefill*: the prompt (system prompt, tool definitions, conversation history, your question) is + processed in one batch, at roughly 600 to 700 tokens per second when nothing is cached. +3. *Decode*: the answer is generated token by token, at 50 to 60 tokens per second for this model. + +The wait before anything appears, the time to first token, is load plus prefill. A cold first question +therefore takes 20 seconds before the first word; a warm one under a second. + +*One question, many requests.* The panel answers by calling the `tui_*` tools, and every tool call the +model makes costs a new request that sends the whole prompt again. A simple question can take 3 to 13 +requests. This is affordable only because Ollama caches the prompt prefix: the system prompt, the tool +definitions and the history are identical from one step to the next, so each step prefills only the new +tokens and takes about half a second. Across a session the cache hit is typically above 90%. The +Ollama tab shows the request count per question as `×N` and the cache hit per question; a question +with a high count and a short answer is the model exploring, which a smaller, sharper tool set reduces +(see <<_tool_set_for_local_models>>). + +*The context window.* The TUI asks Ollama for a 32k context window; `OLLAMA_CONTEXT_LENGTH` overrides +it. The static prefix of system prompt and core tools is about 4.5k tokens, 14% of the window, and each +question with tool calls adds another 2k to 4k of history. The panel compacts the history once it +grows past roughly 16k tokens (see `/compact` under <<_ai_panel_slash_commands>>). Compacting rewrites +the history, which invalidates Ollama's prompt cache, so the request after a compaction prefills the +whole prompt again: about 40 seconds for a 21k-token prompt. That is why the panel compacts local +history late and rarely rather than a little on every turn. Two practical rules follow: + +* Keep every Ollama client on the same context size. A request with a different `num_ctx` makes + Ollama reload the model, which costs a cold start. The TUI's AI panel, `camel ask` and your own routes + should agree on the value. +* A bigger window is cheap in memory for mixture-of-experts models with hybrid attention (this model + grows from 22.5 GB at 32k to 23.6 GB at 262k) but every token in it is prefill time after a + compaction or a reload, so raise it with a purpose. + +*Choosing the model.* With no `camel.tui.ai.model` set the panel takes `llama3.2` when it is installed, +otherwise the first installed model; run `/model <name>` in the panel or set *AI Model* in +*F2 -> Settings* to pin one. The model must support tool calling and should have at least 14B +parameters; a mixture-of-experts model such as `qwen3.6:35b-a3b` prefills several times faster than a +dense model of similar quality, which is what matters for a tool-heavy prompt. *F2 -> Run Doctor* shows +whether Ollama was found, which models are installed and whether they are large enough. + +*Where to look.* The <<_ollama>> tab is the instrument for all of the above: tokens per second live and +per request, time to first token with cold starts marked, cache hit, how full the context window is +and how it grows per question, GPU and process load, and one line per question with the requests it +took. In the AI panel, `/context` prints what the next request will cost, `/usage` and *Ctrl+U* the +session totals per question. + +*Remote and containerised Ollama.* Everything above applies to an Ollama on another host or inside +`camel infra run ollama` as well, with two differences: the container runs without GPU acceleration, +and the live runner state and host load on the Ollama tab need the server on the same machine. + +For the wider picture see the blog posts +https://camel.apache.org/blog/2026/09/camel-local-model-benchmark/index.html[We had a frontier AI coach a small local model through Camel] +on what a local model can do with Camel and what was changed to help it, and +https://camel.apache.org/blog/2026/09/camel-genai-observability-jbang/index.html[Observe Your Camel AI Routes with GenAI OpenTelemetry] +on observing routes that call Ollama. + ==== Using an OpenAI-compatible local server Set `LLM_API_KEY` and `LLM_BASE_URL` to connect to any OpenAI-compatible server
