[
https://issues.apache.org/jira/browse/CAMEL-27131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146297#comment-18146297
]
Federico Mariani edited comment on CAMEL-27131 at 11/17/26, 4:14 PM:
---------------------------------------------------------------------
The before/after screenshots are now in the issue description, where they
render (this comment embedded them as thumbnails, which this JIRA does not
generate).
_Claude Code on behalf of Croway_
was (Author: JIRAUSER298805):
What a Camel AI route looks like in a tracing backend that reads the
OpenTelemetry GenAI conventions, before and after this change. The backend is a
self-hosted Langfuse 4.81 receiving the spans of {{camel-opentelemetry2}} over
OTLP. The route classifies a support question with {{openai}} and hands order
questions to a {{langchain4j-agent}} that calls an {{ai-tool}} route
({{getOrderStatus}}). The model is a local Qwen3-4B, so no cost is shown.
h3. Before
!1-before-camel-agent-run.png|thumbnail!
Today: the agent run is one {{generate_content}} generation with 469 tokens,
the sum of its two model calls. The tool is a plain {{ai-tool}} span, and no
observation has an input or an output.
!2-before-otel-agent-duplicates.png|thumbnail!
With {{camel-observability-services}} and the OpenTelemetry Java agent: each
{{openai}} call is recorded twice, once by Camel (under the {{openai}} span)
and once by the agent's {{openai-java}} instrumentation (its sibling), so its
tokens are counted twice.
h3. After ({{camel.aiObservability.level=full}})
!3-after-full-model-call.png|thumbnail!
The agent run is an {{invoke_agent}} span with one generation per model call.
The first one shows the user message and the {{getOrderStatus}} tool call that
the model asked for.
!4-after-full-tool-call.png|thumbnail!
The tool call is a tool observation, with its arguments and its result.
!6-after-quarkus-langchain4j-deferred.png|thumbnail!
Quarkus with a quarkus-langchain4j model, at {{level=detailed}}:
quarkus-langchain4j already traces the model calls ({{completion ...}}), so
Camel does not record them again. Its agent run and its route tool hold them in
one tree.
In the Camel Main "after" traces, the trace-level session and user come from
the demo route, not from this change.
_Claude Code on behalf of Croway_
> camel-ai-observability - observability levels with agent, tool and message
> data, and no double recording of calls traced by AI frameworks
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-27131
> URL: https://issues.apache.org/jira/browse/CAMEL-27131
> Project: Camel
> Issue Type: Sub-task
> Components: camel-ai, camel-langchain4j-agent, camel-openai,
> camel-spring-ai, camel-telemetry
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
> Attachments: 1-before-camel-agent-run.png,
> 2-before-otel-agent-duplicates.png, 3-after-full-model-call.png,
> 4-after-full-tool-call.png, 6-after-quarkus-langchain4j-deferred.png
>
>
> Today the AI producers record one span per LLM call with the model, the token
> usage and the finish reason, the stable subset chosen in CAMEL-23861. Tracing
> backends built around LLM traces, which read the OpenTelemetry GenAI semantic
> conventions (for example Langfuse), need more to show what an AI route did,
> and several common setups record each call twice.
> Tested end to end with a self-hosted Langfuse on Camel Main, Spring Boot 4.1
> and Quarkus 4, with {{openai}}, {{langchain4j-agent}} and an {{ai-tool}}
> route, {{spring-ai-chat}} and {{langchain4j-chat}}:
> * no prompt or completion is recorded on any LLM call;
> * a {{langchain4j-agent}} run is one {{generate_content}} span that sums
> every model call of the tool loop, and the tool calls are not traced as tools;
> * the same call is recorded twice, with its tokens, by the OpenTelemetry Java
> agent (its {{openai-java}} instrumentation), by Spring AI ({{ChatModel}}
> observations) and by quarkus-langchain4j ({{SpanChatModelListener}});
> * with both {{camel-opentelemetry2}} and a Micrometer {{ObservationRegistry}}
> (Spring Boot), the Micrometer path is taken: it drops the token usage, the
> response model and the finish reason, and records nothing when no tracing
> handler is registered;
> * {{camel-observability-services}} does not include
> {{camel-ai-observability}}.
> Proposal, keeping today's output as the default:
> * {{camel.aiObservability.level}} = {{basic}} (default, unchanged) |
> {{detailed}} | {{full}}
> ** {{detailed}}: an {{invoke_agent}} span per {{langchain4j-agent}} run and
> per {{openai}} tool loop, one {{chat}} span per model call, an
> {{execute_tool}} span per tool call (Camel route tools through
> {{AiToolExecutor}}, MCP tools of {{openai}}), request parameters, cached
> tokens, {{gen_ai.provider.name}} and {{gen_ai.conversation.id}};
> ** {{full}}: {{gen_ai.input.messages}}, {{gen_ai.output.messages}},
> {{gen_ai.system_instructions}}, tool arguments and results, each text cut at
> {{camel.aiObservability.contentMaxLength}} (8192).
> * {{camel.aiObservability.deferToFramework}} (default true): no Camel span
> for the calls that Spring AI, quarkus-langchain4j, a
> {{langchain4j-observation}} listener or the OpenTelemetry Java agent already
> trace.
> * Metrics are the same at every level. Agent runs and tool calls recorded
> through Micrometer are named {{camel.ai.agent}} and {{camel.ai.tool}}, apart
> from {{gen_ai.client.operation}}.
> * At every level: numeric token attributes, the usage as high-cardinality
> values of the Micrometer Observation (as Spring AI does), GenAI spans through
> {{camel-opentelemetry2}} when it is in use, and GenAI spans made current
> during synchronous calls so that the spans of the HTTP client or the AI
> framework nest under them.
> * {{camel-observability-services}} depends on {{camel-ai-observability}}.
> h3. Before and after
> What a Camel AI route looks like in a tracing backend that reads the
> OpenTelemetry GenAI conventions (here a self-hosted Langfuse 4.81 receiving
> the spans of {{camel-opentelemetry2}} over OTLP). The route classifies a
> support question with {{openai}} and hands order questions to a
> {{langchain4j-agent}} that calls an {{ai-tool}} route ({{getOrderStatus}}).
> The model is a local Qwen3-4B, so no cost is shown.
> *Before.* The agent run is one {{generate_content}} generation with 469
> tokens, the sum of its two model calls. The tool is a plain {{ai-tool}} span,
> and no observation has an input or an output.
> !1-before-camel-agent-run.png|width=900!
> *Before, with {{camel-observability-services}} and the OpenTelemetry Java
> agent.* Each {{openai}} call is recorded twice, once by Camel (under the
> {{openai}} span) and once by the agent's {{openai-java}} instrumentation (its
> sibling), so its tokens are counted twice.
> !2-before-otel-agent-duplicates.png|width=900!
> *After, {{camel.aiObservability.level=full}}.* The agent run is an
> {{invoke_agent}} span with one generation per model call. The first one shows
> the user message and the {{getOrderStatus}} tool call that the model asked
> for.
> !3-after-full-model-call.png|width=900!
> *After.* The tool call is a tool observation, with its arguments and its
> result.
> !4-after-full-tool-call.png|width=900!
> *After, Quarkus with a quarkus-langchain4j model, {{level=detailed}}.*
> quarkus-langchain4j already traces the model calls ({{completion ...}}), so
> Camel does not record them again. Camel's agent run and route tool hold them
> in one tree.
> !6-after-quarkus-langchain4j-deferred.png|width=900!
> In the Camel Main "after" traces, the trace-level session and user come from
> the demo route, not from this change.
> The camel-spring-boot and camel-quarkus parts are tracked separately.
> _Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)