Claus Ibsen created CAMEL-24796:
-----------------------------------
Summary: camel-jbang - TUI AI panel with Ollama: compact on
measured tokens, compact harder, and raise the 32k default context
Key: CAMEL-24796
URL: https://issues.apache.org/jira/browse/CAMEL-24796
Project: Camel
Issue Type: Improvement
Components: camel-jbang
Reporter: Claus Ibsen
Observed with the new Ollama tab (CAMEL-24794) while using the TUI AI panel
(F8) against Ollama 0.33.3 serving qwen3.6:35b-a3b on an Apple M4 Pro, 64 GB.
Six questions with tool calls produced 54 Ollama requests. The prompt grew from
4.5k tokens (system prompt plus core tools plus question) to 24.8k, 76% of the
32k window the client asks for, before the panel compacted. The compaction then
freed only 3.8k tokens (down to 21.0k, 64%) and, because rewriting history
invalidates Ollama's prompt cache, the next request re-evaluated all 21k tokens
at 528 tok/s: about 40 s time to first token against 0.3 to 0.7 s for the
cached steps before it.
Three changes to the AI panel (AiPanel.compactHistoryAfterTurn /
shouldCompactAfterTurn / compactHistory in camel-jbang-plugin-tui):
1. *Trigger on measured tokens, not an estimate.* The panel compacts when its
character-based estimate of the history (chars / 4) passes
LOCAL_HISTORY_BUDGET_TOKENS = 16000. Tool results are JSON and tokenize nearer
3 chars per token, so the real prompt was 24.8k when the estimate crossed 16k.
Ollama reports the exact prompt size (prompt_eval_count, now in
LlmClient.TokenUsage.inputTokens together with cached, prefill, generation,
load and total) on every reply; use that against the num_ctx the client
requested, and compact at about half the window.
2. *Compact big when compacting.* Today compaction shrinks tool results of
older turns to 400 characters and drops turns only beyond 20. Locally every
compaction costs a full re-prefill, so it must be worth it: keep the last two
or three turns intact, reduce older turns to question plus final answer, and
aim for the prompt to land around 25 to 30% of the window rather than 64%.
Consider doing it once at the threshold rather than a little every turn. Show
"compacted, the next reply prefills the whole prompt" in the panel and the
measured fill (e.g. ctx 64%) in the panel title, so the climb is visible before
it hurts.
3. *Tune the default context size.* LlmClient.OLLAMA_NUM_CTX = 32768 (override
with OLLAMA_CONTEXT_LENGTH) is low for a tool-calling panel: the static prefix
alone is 4.5k and each question with tool calls adds 2 to 4k. Measured cost of
a bigger window on this model: 22.47 GB loaded at 32k versus 23.57 GB at 262k
(the hybrid attention of qwen3.5 MoE keeps the KV cache small), so 64k or 128k
is affordable on typical developer machines; dense models cost more per token
of context, so the default should probably depend on the model's max context
(from /api/show) and available memory, capped around 64k or 128k, with the
prefill speed in mind (a full re-prefill of 128k at 600 tok/s is 3.5 minutes).
Whatever value is chosen must be the same for every Ollama request the TUI
makes, because a differing num_ctx makes Ollama reload the model (20 s cold
start observed).
The Ollama tab (CTX column, turns line, CACHED column) and tui_get_ollama make
the effect of these changes measurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
