[ 
https://issues.apache.org/jira/browse/CAMEL-27131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Federico Mariani updated CAMEL-27131:
-------------------------------------
    Fix Version/s: 4.24.0

> camel-ai-observability - observability levels with agent, tool and message 
> data, and no double recording of calls traced by AI frameworks
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27131
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27131
>             Project: Camel
>          Issue Type: Sub-task
>          Components: camel-ai, camel-langchain4j-agent, camel-openai, 
> camel-spring-ai, camel-telemetry
>            Reporter: Federico Mariani
>            Assignee: Federico Mariani
>            Priority: Major
>             Fix For: 4.24.0
>
>         Attachments: 1-before-camel-agent-run.png, 
> 2-before-otel-agent-duplicates.png, 3-after-full-model-call.png, 
> 4-after-full-tool-call.png, 6-after-quarkus-langchain4j-deferred.png, 
> camel-27131-perf-results.jsonl
>
>
> Today the AI producers record one span per LLM call with the model, the token 
> usage and the finish reason, the stable subset chosen in CAMEL-23861. Tracing 
> backends built around LLM traces, which read the OpenTelemetry GenAI semantic 
> conventions (for example Langfuse), need more to show what an AI route did, 
> and several common setups record each call twice.
> Tested end to end with a self-hosted Langfuse on Camel Main, Spring Boot 4.1 
> and Quarkus 4, with {{openai}}, {{langchain4j-agent}} and an {{ai-tool}} 
> route, {{spring-ai-chat}} and {{langchain4j-chat}}:
> * no prompt or completion is recorded on any LLM call;
> * a {{langchain4j-agent}} run is one {{generate_content}} span that sums 
> every model call of the tool loop, and the tool calls are not traced as tools;
> * the same call is recorded twice, with its tokens, by the OpenTelemetry Java 
> agent (its {{openai-java}} instrumentation), by Spring AI ({{ChatModel}} 
> observations) and by quarkus-langchain4j ({{SpanChatModelListener}});
> * with both {{camel-opentelemetry2}} and a Micrometer {{ObservationRegistry}} 
> (Spring Boot), the Micrometer path is taken: it drops the token usage, the 
> response model and the finish reason, and records nothing when no tracing 
> handler is registered;
> * {{camel-observability-services}} does not include 
> {{camel-ai-observability}}.
> Proposal, keeping today's output as the default:
> * {{camel.aiObservability.level}} = {{basic}} (default, unchanged) | 
> {{detailed}} | {{full}}
> ** {{detailed}}: an {{invoke_agent}} span per {{langchain4j-agent}} run and 
> per {{openai}} tool loop, one {{chat}} span per model call, an 
> {{execute_tool}} span per tool call (Camel route tools through 
> {{AiToolExecutor}}, MCP tools of {{openai}}), request parameters, cached 
> tokens, {{gen_ai.provider.name}} and {{gen_ai.conversation.id}};
> ** {{full}}: {{gen_ai.input.messages}}, {{gen_ai.output.messages}}, 
> {{gen_ai.system_instructions}}, tool arguments and results, each text cut at 
> {{camel.aiObservability.contentMaxLength}} (8192).
> * {{camel.aiObservability.deferToFramework}} (default true): no Camel span 
> for the calls that Spring AI, quarkus-langchain4j, a 
> {{langchain4j-observation}} listener or the OpenTelemetry Java agent already 
> trace.
> * Metrics are the same at every level. Agent runs and tool calls recorded 
> through Micrometer are named {{camel.ai.agent}} and {{camel.ai.tool}}, apart 
> from {{gen_ai.client.operation}}.
> * At every level: numeric token attributes, the usage as high-cardinality 
> values of the Micrometer Observation (as Spring AI does), GenAI spans through 
> {{camel-opentelemetry2}} when it is in use, and GenAI spans made current 
> during synchronous calls so that the spans of the HTTP client or the AI 
> framework nest under them.
> * {{camel-observability-services}} depends on {{camel-ai-observability}}.
> h3. Before and after
> What a Camel AI route looks like in a tracing backend that reads the 
> OpenTelemetry GenAI conventions (here a self-hosted Langfuse 4.81 receiving 
> the spans of {{camel-opentelemetry2}} over OTLP). The route classifies a 
> support question with {{openai}} and hands order questions to a 
> {{langchain4j-agent}} that calls an {{ai-tool}} route ({{getOrderStatus}}). 
> The model is a local Qwen3-4B, so no cost is shown.
> *Before.* The agent run is one {{generate_content}} generation with 469 
> tokens, the sum of its two model calls. The tool is a plain {{ai-tool}} span, 
> and no observation has an input or an output.
> !1-before-camel-agent-run.png|width=900!
> *Before, with {{camel-observability-services}} and the OpenTelemetry Java 
> agent.* Each {{openai}} call is recorded twice, once by Camel (under the 
> {{openai}} span) and once by the agent's {{openai-java}} instrumentation (its 
> sibling), so its tokens are counted twice.
> !2-before-otel-agent-duplicates.png|width=900!
> *After, {{camel.aiObservability.level=full}}.* The agent run is an 
> {{invoke_agent}} span with one generation per model call. The first one shows 
> the user message and the {{getOrderStatus}} tool call that the model asked 
> for.
> !3-after-full-model-call.png|width=900!
> *After.* The tool call is a tool observation, with its arguments and its 
> result.
> !4-after-full-tool-call.png|width=900!
> *After, Quarkus with a quarkus-langchain4j model, {{level=detailed}}.* 
> quarkus-langchain4j already traces the model calls ({{completion ...}}), so 
> Camel does not record them again. Camel's agent run and route tool hold them 
> in one tree.
> !6-after-quarkus-langchain4j-deferred.png|width=900!
> In the Camel Main "after" traces, the trace-level session and user come from 
> the demo route, not from this change.
> The camel-spring-boot and camel-quarkus parts are tracked separately.
> _Claude Code on behalf of Croway_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to