davsclaus commented on code in PR #26550:
URL: https://github.com/apache/camel/pull/26550#discussion_r4037640900
##########
dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/LlmClient.java:
##########
@@ -999,11 +1014,64 @@ private ChatResponse chatOpenAiFormat(String
systemPrompt, List<Message> message
private JsonObject ollamaOptions() {
JsonObject options = new JsonObject();
options.put("temperature", temperature);
- options.put("num_ctx", ollamaNumCtx());
+ options.put("num_ctx", ollamaContextWindow());
return options;
}
- static int ollamaNumCtx() {
+ // ---- Ollama context window ----
+
+ private Integer resolvedOllamaContext;
+ private String resolvedOllamaContextModel;
+ private String resolvedOllamaContextUrl;
+
+ /**
+ * The context window ({@code num_ctx}) this client asks Ollama for with
the current model, resolved once per model
+ * and endpoint:
+ * <ol>
+ * <li>{@code OLLAMA_CONTEXT_LENGTH} in the environment wins when set.</li>
+ * <li>If Ollama already has the model loaded, its window is adopted
(raised to {@link #OLLAMA_MIN_CONTEXT} when
+ * smaller), because a request with a different {@code num_ctx} makes
Ollama reload the model, which costs a cold
+ * start and throws away the prompt cache of every other client.</li>
+ * <li>Otherwise {@link #OLLAMA_MAX_CONTEXT} when the model's weights plus
the KV cache of that window fit the
+ * machine's memory, else {@link #OLLAMA_MIN_CONTEXT}. Overshooting is
worse than being conservative: Ollama then
+ * moves layers to the CPU and generation slows to a crawl.</li>
+ * </ol>
+ * Callers that manage a conversation should budget their history against
{@code min(window, OLLAMA_MAX_CONTEXT)}
+ * even when a larger window was adopted, so a cache loss never means
minutes of prefill.
+ */
+ public synchronized int ollamaContextWindow() {
+ if (resolvedOllamaContext != null &&
Objects.equals(resolvedOllamaContextModel, model)
+ && Objects.equals(resolvedOllamaContextUrl, url)) {
+ return resolvedOllamaContext;
+ }
+ int window = resolveOllamaContextWindow(totalPhysicalMemory());
Review Comment:
Fixed in af10968db134: the window is now resolved on a daemon thread when
the panel connects to an Ollama client and again after `/model`, so the cache
is warm before the first question
(`LlmClient.resolveOllamaContextWindowInBackground()`, covered by
`theWindowCanBeResolvedAheadOfTheFirstRequest`). A request that arrives earlier
still resolves synchronously, as before.
For scale: the three probes go to the same local server that answers the
chat and take milliseconds; the 5 s timeouts only apply when Ollama is
unreachable, and then the chat request fails as well. The stall was real in
shape but small in practice; it is gone either way.
_Claude Code on behalf of davsclaus_
##########
dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/LlmClient.java:
##########
@@ -1012,10 +1080,155 @@ static int ollamaNumCtx() {
return value;
}
} catch (NumberFormatException e) {
- // fall through to the default
+ // fall through to the policy
+ }
+ }
+ return null;
+ }
+
+ /** Context length of the current model if Ollama already has it loaded
({@code /api/ps}), 0 otherwise. */
+ long loadedOllamaContext() {
+ JsonObject ps = sendGetRequest(url + "/api/ps", Map.of());
+ if (ps == null) {
+ return 0;
+ }
+ Collection<?> loaded = ps.getCollection("models");
+ if (loaded == null) {
+ return 0;
+ }
+ for (Object o : loaded) {
+ if (o instanceof JsonObject m && ollamaModelMatches(model,
m.getString("name"))) {
+ return getLongValue(m, "context_length");
}
}
- return OLLAMA_NUM_CTX;
+ return 0;
+ }
+
+ /** {@code llama3.2} matches {@code llama3.2:latest}; a name with a tag
must match exactly. */
+ static boolean ollamaModelMatches(String wanted, String candidate) {
+ if (wanted == null || candidate == null) {
+ return false;
+ }
+ if (wanted.equals(candidate)) {
+ return true;
+ }
+ return !wanted.contains(":") && candidate.startsWith(wanted + ":");
+ }
+
+ /**
+ * Whether the current model's weights plus the KV cache of a {@code
contextTokens} window fit in
+ * {@code totalMemoryBytes}: weights from {@code /api/tags}, the KV cost
per token from the architecture facts in
+ * {@code /api/show}. Unknown facts count as "does not fit".
+ */
+ boolean ollamaContextFits(int contextTokens, long totalMemoryBytes) {
+ if (totalMemoryBytes <= 0) {
+ return false;
+ }
+ long weights = ollamaWeightBytes();
+ if (weights <= 0) {
+ return false;
+ }
+ JsonObject body = new JsonObject();
+ body.put("model", model);
+ JsonObject show = postJsonQuietly(url + "/api/show", body);
+ long perToken = ollamaKvBytesPerToken(show);
+ if (perToken <= 0) {
+ return false;
+ }
+ return ollamaContextFits(weights, perToken, contextTokens,
totalMemoryBytes);
+ }
+
+ /**
+ * Four fifths of the memory less a gibibyte for compute buffers: roughly
what Apple silicon lets the GPU wire and a
+ * fair share of a machine that also runs the integration and the TUI. A
heuristic, deliberately on the safe side.
+ */
+ static boolean ollamaContextFits(long weightBytes, long kvBytesPerToken,
int contextTokens, long totalMemoryBytes) {
+ long budget = totalMemoryBytes / 5 * 4 - (1L << 30);
+ return weightBytes + kvBytesPerToken * contextTokens <= budget;
+ }
+
+ /**
+ * Bytes of KV cache per context token: 16-bit keys and values for each KV
head of each layer that keeps a full
+ * attention cache. Hybrid models (Qwen 3.5/3.6 MoE) report {@code
full_attention_interval}; only every n-th layer
+ * holds a cache, the others carry a fixed-size state. Head dimensions
come from {@code attention.key_length} and
+ * {@code attention.value_length} when reported, else from the embedding
size over the head count.
+ */
+ static long ollamaKvBytesPerToken(JsonObject show) {
+ if (show == null || !(show.get("model_info") instanceof JsonObject
info)) {
+ return 0;
+ }
+ String arch = info.getString("general.architecture");
+ if (arch == null) {
+ return 0;
+ }
+ long layers = getLongValue(info, arch + ".block_count");
+ long heads = getLongValue(info, arch + ".attention.head_count");
+ long kvHeads = getLongValue(info, arch + ".attention.head_count_kv");
+ long embedding = getLongValue(info, arch + ".embedding_length");
+ long keyLength = getLongValue(info, arch + ".attention.key_length");
+ long valueLength = getLongValue(info, arch +
".attention.value_length");
+ long interval = getLongValue(info, arch + ".full_attention_interval");
+ if (kvHeads <= 0) {
+ kvHeads = heads;
+ }
+ if (keyLength <= 0) {
+ keyLength = heads > 0 ? embedding / heads : 0;
+ }
+ if (valueLength <= 0) {
+ valueLength = keyLength;
+ }
+ long cacheLayers = interval > 1 ? layers / interval : layers;
+ if (cacheLayers <= 0 || kvHeads <= 0 || keyLength <= 0) {
+ return 0;
+ }
+ return cacheLayers * kvHeads * (keyLength + valueLength) * 2L;
+ }
+
+ /** A short POST with the health-check timeout and no console output, for
probes outside a chat request. */
+ private JsonObject postJsonQuietly(String requestUrl, JsonObject body) {
+ try {
+ HttpRequest request = HttpRequest.newBuilder()
+ .uri(URI.create(requestUrl))
+ .timeout(Duration.ofSeconds(HEALTH_CHECK_TIMEOUT_SECONDS))
+ .header("Content-Type", "application/json")
+ .POST(HttpRequest.BodyPublishers.ofString(body.toJson()))
+ .build();
+ HttpResponse<String> response = httpClient.send(request,
HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() == 200) {
+ return (JsonObject) Jsoner.deserialize(response.body());
+ }
+ } catch (Exception e) {
+ // the caller treats an unknown answer as "does not fit"
+ }
+ return null;
+ }
+
+ private long ollamaWeightBytes() {
+ JsonObject tags = sendGetRequest(url + "/api/tags", Map.of());
+ if (tags == null) {
+ return 0;
+ }
+ Collection<?> models = tags.getCollection("models");
+ if (models == null) {
+ return 0;
+ }
+ for (Object o : models) {
+ if (o instanceof JsonObject m && ollamaModelMatches(model,
m.getString("name"))) {
+ return getLongValue(m, "size");
+ }
+ }
+ return 0;
+ }
+
+ static long totalPhysicalMemory() {
+ try {
+ if (ManagementFactory.getOperatingSystemMXBean() instanceof
OperatingSystemMXBean os) {
Review Comment:
Keeping this one. `com.sun.management` is a supported, exported API of the
`jdk.management` module (it is listed in the module's `exports`, not
`jdk.internal.*`), so `--release 17` compiles it without a warning and there is
no `restriction` category to suppress in javac. Camel already uses the same
interface in production code: `HeapDumpDevConsole` and
`HeapHistogramDevConsole` in camel-console. The `instanceof` plus `Throwable`
guard already covers JVMs that do not offer it (the fit check then falls back
to the 32k minimum).
_Claude Code on behalf of davsclaus_
##########
dsl/camel-jbang/camel-jbang-plugin-tui/src/main/java/org/apache/camel/dsl/jbang/core/commands/tui/AiPanel.java:
##########
@@ -1996,11 +2208,26 @@ private void recordUsage(LlmClient.ChatResponse
response, long latencyMs) {
response.usage().totalTokens(), latencyMs,
response.stopReason(), Instant.now(),
AiUsageSource.TUI, null, questionCounter));
+ noteMeasuredPrompt(response.usage());
if (ctx != null && ctx.ollamaMonitor != null && client.apiType() ==
LlmClient.ApiType.ollama) {
// the Ollama tab shows this request with the timings Ollama
returned
ctx.ollamaMonitor.adoptEndpoint(client.endpointUrl());
- ctx.ollamaMonitor.recordRequest(model, response.usage(),
latencyMs, response.stopReason(),
- questionCounter, currentQuestion);
+ ctx.ollamaMonitor.recordRequest(model, response.usage(), latencyMs,
+ reasonOverride != null ? reasonOverride :
response.stopReason(), questionCounter, currentQuestion);
+ ctx.ollamaMonitor.setPanelContext(knownContextWindow,
compactionBudgetTokens(true, knownContextWindow));
+ }
+ }
+
+ /** Keeps what the provider measured: the prompt size, the cold prefill
speed and, for Ollama, the window. */
+ private void noteMeasuredPrompt(LlmClient.TokenUsage usage) {
+ if (usage.inputTokens() > 0) {
+ lastMeasuredPromptTokens = usage.inputTokens();
+ }
+ if (usage.cachedTokens() == 0 && usage.inputTokens() >= 1000 &&
usage.prefillMillis() > 0) {
+ coldPrefillTokensPerSecond = usage.inputTokens() * 1000.0 /
usage.prefillMillis();
+ }
+ if (client != null && client.apiType() == LlmClient.ApiType.ollama) {
+ knownContextWindow = client.ollamaContextWindow();
Review Comment:
Done in af10968db134: the resolved window, model and URL now live in one
immutable record behind a `volatile` field, read outside the lock;
`ollamaContextWindow()` only synchronizes when it has to resolve, so the
per-response call from `noteMeasuredPrompt` is a volatile read.
_Claude Code on behalf of davsclaus_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
