[
https://issues.apache.org/jira/browse/CAMEL-27131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146449#comment-18146449
]
Federico Mariani commented on CAMEL-27131:
------------------------------------------
h3. Cost of the observability levels
Measured on the PR branch (commit 22f4abf103) against a mock LLM, to see what
each level costs in Camel itself.
*Setup*
* A mock OpenAI API (Camel routes, separate JVM) answers instantly with canned
completions, including a tool call for the agent. An OTLP sink in the same JVM
counts the exported bytes. No real LLM, no tracing backend.
* The application runs in a fresh JVM per configuration (1 GB heap, 4 calling
threads, warmup first), with {{camel-opentelemetry2}} exporting over OTLP/HTTP
and {{disableCoreProcessors=true}}.
* Configurations: no tracing; tracing with AI observability off; {{basic}};
{{detailed}}; {{full}}.
* Scenarios: one {{openai}} chat call (0.6 KB system prompt, 0.3 KB question,
0.8 KB answer); one {{langchain4j-agent}} run (two model calls and one
{{ai-tool}} route); the chat call with a 16 KB question.
* 3 repeats each, 45 runs; medians below. The raw results are attached
({{camel-27131-perf-results.jsonl}}).
*Results*, compared with tracing on and AI observability off:
||Level||Chat call: CPU / allocation per request||Agent run (2 model calls +
tool)||Chat with a 16 KB prompt||Exported per request||
|{{basic}}|+10 µs (+3%) / +6.6 KB|+103 µs (within noise) / +9.8 KB|+60 µs /
+7.1 KB|2.3 KB chat, 1.9 KB agent|
|{{detailed}}|+40 µs (+12%) / +7.2 KB|within noise / +39 KB|+55 µs / +7.2
KB|2.3 KB chat, 3.0 KB agent|
|{{full}}|+48 µs (+14%) / +26 KB|+158 µs (+6%) / +162 KB|+126 µs (+18%) / +100
KB|3.9 KB chat, 6.5 KB agent, 11.7 KB large|
* *CPU and allocation:* allocation is the reliable measure, varying by less
than 1 KB between repeats. CPU varies by up to ±180 µs per agent run, so for
the agent only the {{full}} difference stands out.
* *Response time:* the median latency grows by 0.01 to 0.03 ms per chat call
and 0.1 to 0.3 ms per agent run at {{full}}. With a real model taking hundreds
of milliseconds or more per call, that is well under 0.1% of a request.
* *Throughput:* up to about 12 to 15% lower at {{full}}, only because the
instant mock makes the test CPU-bound; with a real model, the LLM sets the
throughput.
* *Memory:* the heap retained after a full GC is the same at every level (26.4
MB chat, 20.8 MB agent), so no level retains memory. GC runs barely change (2
to 3 per run at {{full}}, 2 otherwise).
* *Tracing itself:* going from no tracing to tracing with AI observability off
already adds 109 µs and 11.5 KB per chat call, and 227 µs per agent run. That
is more than any level adds on top.
* *Where {{full}} spends its memory:* each model call of an agent run records
the whole conversation so far and the tool definitions, as the GenAI
conventions and the tracing backends expect. With the 16 KB prompt, the message
JSON is built before each text is cut at 8192 characters; cutting first could
save part of that if it ever matters.
Caveats: one machine (Apple M1 Pro) with the mock services in a second JVM on
it, other idle services in the background, non-streaming calls only.
_Claude Code on behalf of Croway_
> camel-ai-observability - observability levels with agent, tool and message
> data, and no double recording of calls traced by AI frameworks
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-27131
> URL: https://issues.apache.org/jira/browse/CAMEL-27131
> Project: Camel
> Issue Type: Sub-task
> Components: camel-ai, camel-langchain4j-agent, camel-openai,
> camel-spring-ai, camel-telemetry
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
> Attachments: 1-before-camel-agent-run.png,
> 2-before-otel-agent-duplicates.png, 3-after-full-model-call.png,
> 4-after-full-tool-call.png, 6-after-quarkus-langchain4j-deferred.png,
> camel-27131-perf-results.jsonl
>
>
> Today the AI producers record one span per LLM call with the model, the token
> usage and the finish reason, the stable subset chosen in CAMEL-23861. Tracing
> backends built around LLM traces, which read the OpenTelemetry GenAI semantic
> conventions (for example Langfuse), need more to show what an AI route did,
> and several common setups record each call twice.
> Tested end to end with a self-hosted Langfuse on Camel Main, Spring Boot 4.1
> and Quarkus 4, with {{openai}}, {{langchain4j-agent}} and an {{ai-tool}}
> route, {{spring-ai-chat}} and {{langchain4j-chat}}:
> * no prompt or completion is recorded on any LLM call;
> * a {{langchain4j-agent}} run is one {{generate_content}} span that sums
> every model call of the tool loop, and the tool calls are not traced as tools;
> * the same call is recorded twice, with its tokens, by the OpenTelemetry Java
> agent (its {{openai-java}} instrumentation), by Spring AI ({{ChatModel}}
> observations) and by quarkus-langchain4j ({{SpanChatModelListener}});
> * with both {{camel-opentelemetry2}} and a Micrometer {{ObservationRegistry}}
> (Spring Boot), the Micrometer path is taken: it drops the token usage, the
> response model and the finish reason, and records nothing when no tracing
> handler is registered;
> * {{camel-observability-services}} does not include
> {{camel-ai-observability}}.
> Proposal, keeping today's output as the default:
> * {{camel.aiObservability.level}} = {{basic}} (default, unchanged) |
> {{detailed}} | {{full}}
> ** {{detailed}}: an {{invoke_agent}} span per {{langchain4j-agent}} run and
> per {{openai}} tool loop, one {{chat}} span per model call, an
> {{execute_tool}} span per tool call (Camel route tools through
> {{AiToolExecutor}}, MCP tools of {{openai}}), request parameters, cached
> tokens, {{gen_ai.provider.name}} and {{gen_ai.conversation.id}};
> ** {{full}}: {{gen_ai.input.messages}}, {{gen_ai.output.messages}},
> {{gen_ai.system_instructions}}, tool arguments and results, each text cut at
> {{camel.aiObservability.contentMaxLength}} (8192).
> * {{camel.aiObservability.deferToFramework}} (default true): no Camel span
> for the calls that Spring AI, quarkus-langchain4j, a
> {{langchain4j-observation}} listener or the OpenTelemetry Java agent already
> trace.
> * Metrics are the same at every level. Agent runs and tool calls recorded
> through Micrometer are named {{camel.ai.agent}} and {{camel.ai.tool}}, apart
> from {{gen_ai.client.operation}}.
> * At every level: numeric token attributes, the usage as high-cardinality
> values of the Micrometer Observation (as Spring AI does), GenAI spans through
> {{camel-opentelemetry2}} when it is in use, and GenAI spans made current
> during synchronous calls so that the spans of the HTTP client or the AI
> framework nest under them.
> * {{camel-observability-services}} depends on {{camel-ai-observability}}.
> h3. Before and after
> What a Camel AI route looks like in a tracing backend that reads the
> OpenTelemetry GenAI conventions (here a self-hosted Langfuse 4.81 receiving
> the spans of {{camel-opentelemetry2}} over OTLP). The route classifies a
> support question with {{openai}} and hands order questions to a
> {{langchain4j-agent}} that calls an {{ai-tool}} route ({{getOrderStatus}}).
> The model is a local Qwen3-4B, so no cost is shown.
> *Before.* The agent run is one {{generate_content}} generation with 469
> tokens, the sum of its two model calls. The tool is a plain {{ai-tool}} span,
> and no observation has an input or an output.
> !1-before-camel-agent-run.png|width=900!
> *Before, with {{camel-observability-services}} and the OpenTelemetry Java
> agent.* Each {{openai}} call is recorded twice, once by Camel (under the
> {{openai}} span) and once by the agent's {{openai-java}} instrumentation (its
> sibling), so its tokens are counted twice.
> !2-before-otel-agent-duplicates.png|width=900!
> *After, {{camel.aiObservability.level=full}}.* The agent run is an
> {{invoke_agent}} span with one generation per model call. The first one shows
> the user message and the {{getOrderStatus}} tool call that the model asked
> for.
> !3-after-full-model-call.png|width=900!
> *After.* The tool call is a tool observation, with its arguments and its
> result.
> !4-after-full-tool-call.png|width=900!
> *After, Quarkus with a quarkus-langchain4j model, {{level=detailed}}.*
> quarkus-langchain4j already traces the model calls ({{completion ...}}), so
> Camel does not record them again. Camel's agent run and route tool hold them
> in one tree.
> !6-after-quarkus-langchain4j-deferred.png|width=900!
> In the Camel Main "after" traces, the trace-level session and user come from
> the demo route, not from this change.
> The camel-spring-boot and camel-quarkus parts are tracked separately.
> _Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)