[ 
https://issues.apache.org/jira/browse/CAMEL-27131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146297#comment-18146297
 ] 

Federico Mariani edited comment on CAMEL-27131 at 11/17/26, 4:14 PM:
---------------------------------------------------------------------

The before/after screenshots are now in the issue description, where they 
render (this comment embedded them as thumbnails, which this JIRA does not 
generate).

_Claude Code on behalf of Croway_


was (Author: JIRAUSER298805):
What a Camel AI route looks like in a tracing backend that reads the 
OpenTelemetry GenAI conventions, before and after this change. The backend is a 
self-hosted Langfuse 4.81 receiving the spans of {{camel-opentelemetry2}} over 
OTLP. The route classifies a support question with {{openai}} and hands order 
questions to a {{langchain4j-agent}} that calls an {{ai-tool}} route 
({{getOrderStatus}}). The model is a local Qwen3-4B, so no cost is shown.

h3. Before

!1-before-camel-agent-run.png|thumbnail!
Today: the agent run is one {{generate_content}} generation with 469 tokens, 
the sum of its two model calls. The tool is a plain {{ai-tool}} span, and no 
observation has an input or an output.

!2-before-otel-agent-duplicates.png|thumbnail!
With {{camel-observability-services}} and the OpenTelemetry Java agent: each 
{{openai}} call is recorded twice, once by Camel (under the {{openai}} span) 
and once by the agent's {{openai-java}} instrumentation (its sibling), so its 
tokens are counted twice.

h3. After ({{camel.aiObservability.level=full}})

!3-after-full-model-call.png|thumbnail!
The agent run is an {{invoke_agent}} span with one generation per model call. 
The first one shows the user message and the {{getOrderStatus}} tool call that 
the model asked for.

!4-after-full-tool-call.png|thumbnail!
The tool call is a tool observation, with its arguments and its result.

!6-after-quarkus-langchain4j-deferred.png|thumbnail!
Quarkus with a quarkus-langchain4j model, at {{level=detailed}}: 
quarkus-langchain4j already traces the model calls ({{completion ...}}), so 
Camel does not record them again. Its agent run and its route tool hold them in 
one tree.

In the Camel Main "after" traces, the trace-level session and user come from 
the demo route, not from this change.

_Claude Code on behalf of Croway_


> camel-ai-observability - observability levels with agent, tool and message 
> data, and no double recording of calls traced by AI frameworks
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27131
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27131
>             Project: Camel
>          Issue Type: Sub-task
>          Components: camel-ai, camel-langchain4j-agent, camel-openai, 
> camel-spring-ai, camel-telemetry
>            Reporter: Federico Mariani
>            Assignee: Federico Mariani
>            Priority: Major
>         Attachments: 1-before-camel-agent-run.png, 
> 2-before-otel-agent-duplicates.png, 3-after-full-model-call.png, 
> 4-after-full-tool-call.png, 6-after-quarkus-langchain4j-deferred.png
>
>
> Today the AI producers record one span per LLM call with the model, the token 
> usage and the finish reason, the stable subset chosen in CAMEL-23861. Tracing 
> backends built around LLM traces, which read the OpenTelemetry GenAI semantic 
> conventions (for example Langfuse), need more to show what an AI route did, 
> and several common setups record each call twice.
> Tested end to end with a self-hosted Langfuse on Camel Main, Spring Boot 4.1 
> and Quarkus 4, with {{openai}}, {{langchain4j-agent}} and an {{ai-tool}} 
> route, {{spring-ai-chat}} and {{langchain4j-chat}}:
> * no prompt or completion is recorded on any LLM call;
> * a {{langchain4j-agent}} run is one {{generate_content}} span that sums 
> every model call of the tool loop, and the tool calls are not traced as tools;
> * the same call is recorded twice, with its tokens, by the OpenTelemetry Java 
> agent (its {{openai-java}} instrumentation), by Spring AI ({{ChatModel}} 
> observations) and by quarkus-langchain4j ({{SpanChatModelListener}});
> * with both {{camel-opentelemetry2}} and a Micrometer {{ObservationRegistry}} 
> (Spring Boot), the Micrometer path is taken: it drops the token usage, the 
> response model and the finish reason, and records nothing when no tracing 
> handler is registered;
> * {{camel-observability-services}} does not include 
> {{camel-ai-observability}}.
> Proposal, keeping today's output as the default:
> * {{camel.aiObservability.level}} = {{basic}} (default, unchanged) | 
> {{detailed}} | {{full}}
> ** {{detailed}}: an {{invoke_agent}} span per {{langchain4j-agent}} run and 
> per {{openai}} tool loop, one {{chat}} span per model call, an 
> {{execute_tool}} span per tool call (Camel route tools through 
> {{AiToolExecutor}}, MCP tools of {{openai}}), request parameters, cached 
> tokens, {{gen_ai.provider.name}} and {{gen_ai.conversation.id}};
> ** {{full}}: {{gen_ai.input.messages}}, {{gen_ai.output.messages}}, 
> {{gen_ai.system_instructions}}, tool arguments and results, each text cut at 
> {{camel.aiObservability.contentMaxLength}} (8192).
> * {{camel.aiObservability.deferToFramework}} (default true): no Camel span 
> for the calls that Spring AI, quarkus-langchain4j, a 
> {{langchain4j-observation}} listener or the OpenTelemetry Java agent already 
> trace.
> * Metrics are the same at every level. Agent runs and tool calls recorded 
> through Micrometer are named {{camel.ai.agent}} and {{camel.ai.tool}}, apart 
> from {{gen_ai.client.operation}}.
> * At every level: numeric token attributes, the usage as high-cardinality 
> values of the Micrometer Observation (as Spring AI does), GenAI spans through 
> {{camel-opentelemetry2}} when it is in use, and GenAI spans made current 
> during synchronous calls so that the spans of the HTTP client or the AI 
> framework nest under them.
> * {{camel-observability-services}} depends on {{camel-ai-observability}}.
> h3. Before and after
> What a Camel AI route looks like in a tracing backend that reads the 
> OpenTelemetry GenAI conventions (here a self-hosted Langfuse 4.81 receiving 
> the spans of {{camel-opentelemetry2}} over OTLP). The route classifies a 
> support question with {{openai}} and hands order questions to a 
> {{langchain4j-agent}} that calls an {{ai-tool}} route ({{getOrderStatus}}). 
> The model is a local Qwen3-4B, so no cost is shown.
> *Before.* The agent run is one {{generate_content}} generation with 469 
> tokens, the sum of its two model calls. The tool is a plain {{ai-tool}} span, 
> and no observation has an input or an output.
> !1-before-camel-agent-run.png|width=900!
> *Before, with {{camel-observability-services}} and the OpenTelemetry Java 
> agent.* Each {{openai}} call is recorded twice, once by Camel (under the 
> {{openai}} span) and once by the agent's {{openai-java}} instrumentation (its 
> sibling), so its tokens are counted twice.
> !2-before-otel-agent-duplicates.png|width=900!
> *After, {{camel.aiObservability.level=full}}.* The agent run is an 
> {{invoke_agent}} span with one generation per model call. The first one shows 
> the user message and the {{getOrderStatus}} tool call that the model asked 
> for.
> !3-after-full-model-call.png|width=900!
> *After.* The tool call is a tool observation, with its arguments and its 
> result.
> !4-after-full-tool-call.png|width=900!
> *After, Quarkus with a quarkus-langchain4j model, {{level=detailed}}.* 
> quarkus-langchain4j already traces the model calls ({{completion ...}}), so 
> Camel does not record them again. Camel's agent run and route tool hold them 
> in one tree.
> !6-after-quarkus-langchain4j-deferred.png|width=900!
> In the Camel Main "after" traces, the trace-level session and user come from 
> the demo route, not from this change.
> The camel-spring-boot and camel-quarkus parts are tracked separately.
> _Claude Code on behalf of Croway_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to