gnodet-bot commented on code in PR #26550:
URL: https://github.com/apache/camel/pull/26550#discussion_r4037483566
##########
dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/LlmClient.java:
##########
@@ -999,11 +1014,64 @@ private ChatResponse chatOpenAiFormat(String
systemPrompt, List<Message> message
private JsonObject ollamaOptions() {
JsonObject options = new JsonObject();
options.put("temperature", temperature);
- options.put("num_ctx", ollamaNumCtx());
+ options.put("num_ctx", ollamaContextWindow());
return options;
}
- static int ollamaNumCtx() {
+ // ---- Ollama context window ----
+
+ private Integer resolvedOllamaContext;
+ private String resolvedOllamaContextModel;
+ private String resolvedOllamaContextUrl;
+
+ /**
+ * The context window ({@code num_ctx}) this client asks Ollama for with
the current model, resolved once per model
+ * and endpoint:
+ * <ol>
+ * <li>{@code OLLAMA_CONTEXT_LENGTH} in the environment wins when set.</li>
+ * <li>If Ollama already has the model loaded, its window is adopted
(raised to {@link #OLLAMA_MIN_CONTEXT} when
+ * smaller), because a request with a different {@code num_ctx} makes
Ollama reload the model, which costs a cold
+ * start and throws away the prompt cache of every other client.</li>
+ * <li>Otherwise {@link #OLLAMA_MAX_CONTEXT} when the model's weights plus
the KV cache of that window fit the
+ * machine's memory, else {@link #OLLAMA_MIN_CONTEXT}. Overshooting is
worse than being conservative: Ollama then
+ * moves layers to the CPU and generation slows to a crawl.</li>
+ * </ol>
+ * Callers that manage a conversation should budget their history against
{@code min(window, OLLAMA_MAX_CONTEXT)}
+ * even when a larger window was adopted, so a cache loss never means
minutes of prefill.
+ */
+ public synchronized int ollamaContextWindow() {
+ if (resolvedOllamaContext != null &&
Objects.equals(resolvedOllamaContextModel, model)
+ && Objects.equals(resolvedOllamaContextUrl, url)) {
+ return resolvedOllamaContext;
+ }
+ int window = resolveOllamaContextWindow(totalPhysicalMemory());
Review Comment:
⚠️ **Latency spike on first request per model — up to 15 s of synchronous
blocking.**
`ollamaContextWindow()` is `synchronized` and, when the cache is cold, calls
`resolveOllamaContextWindow()`, which makes up to three sequential HTTP
requests (`/api/ps`, `/api/tags`, `/api/show`) each with a 5-second timeout.
The user submits a question; the agent calls `ollamaOptions()` →
`ollamaContextWindow()` → up to 15 s stall before the first `chatWithTools`
call fires. Any concurrent caller (render thread, `noteMeasuredPrompt`) blocks
on the same lock for that whole duration.
This only triggers once per model+url pair (the cache prevents subsequent
calls), but it happens silently before the first token stream starts. Users
will see the spinner stall for up to 15 s with no indication. Consider one of:
1. Resolve the context window eagerly when the model is selected/connected
(before the user asks), so the cache is warm by the time
`ollamaContextWindow()` is called during a request.
2. Or degrade gracefully: if the resolution takes more than 2 s, fall back
to `OLLAMA_MIN_CONTEXT` and let the next request try again.
##########
dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/LlmClient.java:
##########
@@ -1012,10 +1080,155 @@ static int ollamaNumCtx() {
return value;
}
} catch (NumberFormatException e) {
- // fall through to the default
+ // fall through to the policy
+ }
+ }
+ return null;
+ }
+
+ /** Context length of the current model if Ollama already has it loaded
({@code /api/ps}), 0 otherwise. */
+ long loadedOllamaContext() {
+ JsonObject ps = sendGetRequest(url + "/api/ps", Map.of());
+ if (ps == null) {
+ return 0;
+ }
+ Collection<?> loaded = ps.getCollection("models");
+ if (loaded == null) {
+ return 0;
+ }
+ for (Object o : loaded) {
+ if (o instanceof JsonObject m && ollamaModelMatches(model,
m.getString("name"))) {
+ return getLongValue(m, "context_length");
}
}
- return OLLAMA_NUM_CTX;
+ return 0;
+ }
+
+ /** {@code llama3.2} matches {@code llama3.2:latest}; a name with a tag
must match exactly. */
+ static boolean ollamaModelMatches(String wanted, String candidate) {
+ if (wanted == null || candidate == null) {
+ return false;
+ }
+ if (wanted.equals(candidate)) {
+ return true;
+ }
+ return !wanted.contains(":") && candidate.startsWith(wanted + ":");
+ }
+
+ /**
+ * Whether the current model's weights plus the KV cache of a {@code
contextTokens} window fit in
+ * {@code totalMemoryBytes}: weights from {@code /api/tags}, the KV cost
per token from the architecture facts in
+ * {@code /api/show}. Unknown facts count as "does not fit".
+ */
+ boolean ollamaContextFits(int contextTokens, long totalMemoryBytes) {
+ if (totalMemoryBytes <= 0) {
+ return false;
+ }
+ long weights = ollamaWeightBytes();
+ if (weights <= 0) {
+ return false;
+ }
+ JsonObject body = new JsonObject();
+ body.put("model", model);
+ JsonObject show = postJsonQuietly(url + "/api/show", body);
+ long perToken = ollamaKvBytesPerToken(show);
+ if (perToken <= 0) {
+ return false;
+ }
+ return ollamaContextFits(weights, perToken, contextTokens,
totalMemoryBytes);
+ }
+
+ /**
+ * Four fifths of the memory less a gibibyte for compute buffers: roughly
what Apple silicon lets the GPU wire and a
+ * fair share of a machine that also runs the integration and the TUI. A
heuristic, deliberately on the safe side.
+ */
+ static boolean ollamaContextFits(long weightBytes, long kvBytesPerToken,
int contextTokens, long totalMemoryBytes) {
+ long budget = totalMemoryBytes / 5 * 4 - (1L << 30);
+ return weightBytes + kvBytesPerToken * contextTokens <= budget;
+ }
+
+ /**
+ * Bytes of KV cache per context token: 16-bit keys and values for each KV
head of each layer that keeps a full
+ * attention cache. Hybrid models (Qwen 3.5/3.6 MoE) report {@code
full_attention_interval}; only every n-th layer
+ * holds a cache, the others carry a fixed-size state. Head dimensions
come from {@code attention.key_length} and
+ * {@code attention.value_length} when reported, else from the embedding
size over the head count.
+ */
+ static long ollamaKvBytesPerToken(JsonObject show) {
+ if (show == null || !(show.get("model_info") instanceof JsonObject
info)) {
+ return 0;
+ }
+ String arch = info.getString("general.architecture");
+ if (arch == null) {
+ return 0;
+ }
+ long layers = getLongValue(info, arch + ".block_count");
+ long heads = getLongValue(info, arch + ".attention.head_count");
+ long kvHeads = getLongValue(info, arch + ".attention.head_count_kv");
+ long embedding = getLongValue(info, arch + ".embedding_length");
+ long keyLength = getLongValue(info, arch + ".attention.key_length");
+ long valueLength = getLongValue(info, arch +
".attention.value_length");
+ long interval = getLongValue(info, arch + ".full_attention_interval");
+ if (kvHeads <= 0) {
+ kvHeads = heads;
+ }
+ if (keyLength <= 0) {
+ keyLength = heads > 0 ? embedding / heads : 0;
+ }
+ if (valueLength <= 0) {
+ valueLength = keyLength;
+ }
+ long cacheLayers = interval > 1 ? layers / interval : layers;
+ if (cacheLayers <= 0 || kvHeads <= 0 || keyLength <= 0) {
+ return 0;
+ }
+ return cacheLayers * kvHeads * (keyLength + valueLength) * 2L;
+ }
+
+ /** A short POST with the health-check timeout and no console output, for
probes outside a chat request. */
+ private JsonObject postJsonQuietly(String requestUrl, JsonObject body) {
+ try {
+ HttpRequest request = HttpRequest.newBuilder()
+ .uri(URI.create(requestUrl))
+ .timeout(Duration.ofSeconds(HEALTH_CHECK_TIMEOUT_SECONDS))
+ .header("Content-Type", "application/json")
+ .POST(HttpRequest.BodyPublishers.ofString(body.toJson()))
+ .build();
+ HttpResponse<String> response = httpClient.send(request,
HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() == 200) {
+ return (JsonObject) Jsoner.deserialize(response.body());
+ }
+ } catch (Exception e) {
+ // the caller treats an unknown answer as "does not fit"
+ }
+ return null;
+ }
+
+ private long ollamaWeightBytes() {
+ JsonObject tags = sendGetRequest(url + "/api/tags", Map.of());
+ if (tags == null) {
+ return 0;
+ }
+ Collection<?> models = tags.getCollection("models");
+ if (models == null) {
+ return 0;
+ }
+ for (Object o : models) {
+ if (o instanceof JsonObject m && ollamaModelMatches(model,
m.getString("name"))) {
+ return getLongValue(m, "size");
+ }
+ }
+ return 0;
+ }
+
+ static long totalPhysicalMemory() {
+ try {
+ if (ManagementFactory.getOperatingSystemMXBean() instanceof
OperatingSystemMXBean os) {
Review Comment:
⚠️ **`com.sun.management.OperatingSystemMXBean` is a JDK-internal API —
first use in production code in this module.**
The `catch (Throwable)` guard handles the case where the cast doesn't
succeed (non-HotSpot JVMs, GraalVM native image, module access restrictions).
That's the correct defensive pattern and it's fine at runtime. However, this is
a `com.sun.*` import in production source, which:
- Will generate compiler warnings (`--release` mode flags it).
- May fail if the build toolchain enforces `--illegal-access=deny` or a
future JDK restricts access to `jdk.management`.
A cleaner approach is to stay in `java.lang.management` and call
`getTotalPhysicalMemorySize()` reflectively only if the concrete class is
available — or simply keep the current `instanceof` pattern but add a
`@SuppressWarnings("restriction")` annotation and a comment in the Javadoc
noting the intentional use of the internal API.
##########
dsl/camel-jbang/camel-jbang-plugin-tui/src/main/java/org/apache/camel/dsl/jbang/core/commands/tui/AiPanel.java:
##########
@@ -1996,11 +2208,26 @@ private void recordUsage(LlmClient.ChatResponse
response, long latencyMs) {
response.usage().totalTokens(), latencyMs,
response.stopReason(), Instant.now(),
AiUsageSource.TUI, null, questionCounter));
+ noteMeasuredPrompt(response.usage());
if (ctx != null && ctx.ollamaMonitor != null && client.apiType() ==
LlmClient.ApiType.ollama) {
// the Ollama tab shows this request with the timings Ollama
returned
ctx.ollamaMonitor.adoptEndpoint(client.endpointUrl());
- ctx.ollamaMonitor.recordRequest(model, response.usage(),
latencyMs, response.stopReason(),
- questionCounter, currentQuestion);
+ ctx.ollamaMonitor.recordRequest(model, response.usage(), latencyMs,
+ reasonOverride != null ? reasonOverride :
response.stopReason(), questionCounter, currentQuestion);
+ ctx.ollamaMonitor.setPanelContext(knownContextWindow,
compactionBudgetTokens(true, knownContextWindow));
+ }
+ }
+
+ /** Keeps what the provider measured: the prompt size, the cold prefill
speed and, for Ollama, the window. */
+ private void noteMeasuredPrompt(LlmClient.TokenUsage usage) {
+ if (usage.inputTokens() > 0) {
+ lastMeasuredPromptTokens = usage.inputTokens();
+ }
+ if (usage.cachedTokens() == 0 && usage.inputTokens() >= 1000 &&
usage.prefillMillis() > 0) {
+ coldPrefillTokensPerSecond = usage.inputTokens() * 1000.0 /
usage.prefillMillis();
+ }
+ if (client != null && client.apiType() == LlmClient.ApiType.ollama) {
+ knownContextWindow = client.ollamaContextWindow();
Review Comment:
💡 **`ollamaContextWindow()` called from `noteMeasuredPrompt` — triggers the
3-HTTP-call resolution path on *every* response, not just the first.**
Wait — actually the cache short-circuits after the first call. But the
`synchronized` lock is still acquired on every response. Since
`noteMeasuredPrompt` is called from `recordUsage` inside the agent thread's hot
loop, this adds a `synchronized` round-trip per LLM response. Acceptable, but
worth noting that the cache check is inside the `synchronized` block (not a
double-checked locking shortcut with a volatile), so every call takes the lock
even when the value is cached.
Low-priority but: consider making `resolvedOllamaContext` `volatile` and
doing the null check outside the `synchronized` block for a cheap fast path
(standard safe-publication pattern for immutable-once-set values).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
