[ 
https://issues.apache.org/jira/browse/CAMEL-27131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Federico Mariani updated CAMEL-27131:
-------------------------------------
    Description: 
Today the AI producers record one span per LLM call with the model, the token 
usage and the finish reason, the stable subset chosen in CAMEL-23861. Tracing 
backends built around LLM traces, which read the OpenTelemetry GenAI semantic 
conventions (for example Langfuse), need more to show what an AI route did, and 
several common setups record each call twice.

Tested end to end with a self-hosted Langfuse on Camel Main, Spring Boot 4.1 
and Quarkus 4, with {{openai}}, {{langchain4j-agent}} and an {{ai-tool}} route, 
{{spring-ai-chat}} and {{langchain4j-chat}}:
* no prompt or completion is recorded on any LLM call;
* a {{langchain4j-agent}} run is one {{generate_content}} span that sums every 
model call of the tool loop, and the tool calls are not traced as tools;
* the same call is recorded twice, with its tokens, by the OpenTelemetry Java 
agent (its {{openai-java}} instrumentation), by Spring AI ({{ChatModel}} 
observations) and by quarkus-langchain4j ({{SpanChatModelListener}});
* with both {{camel-opentelemetry2}} and a Micrometer {{ObservationRegistry}} 
(Spring Boot), the Micrometer path is taken: it drops the token usage, the 
response model and the finish reason, and records nothing when no tracing 
handler is registered;
* {{camel-observability-services}} does not include {{camel-ai-observability}}.

Proposal, keeping today's output as the default:
* {{camel.aiObservability.level}} = {{basic}} (default, unchanged) | 
{{detailed}} | {{full}}
** {{detailed}}: an {{invoke_agent}} span per {{langchain4j-agent}} run and per 
{{openai}} tool loop, one {{chat}} span per model call, an {{execute_tool}} 
span per tool call (Camel route tools through {{AiToolExecutor}}, MCP tools of 
{{openai}}), request parameters, cached tokens, {{gen_ai.provider.name}} and 
{{gen_ai.conversation.id}};
** {{full}}: {{gen_ai.input.messages}}, {{gen_ai.output.messages}}, 
{{gen_ai.system_instructions}}, tool arguments and results, each text cut at 
{{camel.aiObservability.contentMaxLength}} (8192).
* {{camel.aiObservability.deferToFramework}} (default true): no Camel span for 
the calls that Spring AI, quarkus-langchain4j, a {{langchain4j-observation}} 
listener or the OpenTelemetry Java agent already trace.
* Metrics are the same at every level. Agent runs and tool calls recorded 
through Micrometer are named {{camel.ai.agent}} and {{camel.ai.tool}}, apart 
from {{gen_ai.client.operation}}.
* At every level: numeric token attributes, the usage as high-cardinality 
values of the Micrometer Observation (as Spring AI does), GenAI spans through 
{{camel-opentelemetry2}} when it is in use, and GenAI spans made current during 
synchronous calls so that the spans of the HTTP client or the AI framework nest 
under them.
* {{camel-observability-services}} depends on {{camel-ai-observability}}.

h3. Before and after

What a Camel AI route looks like in a tracing backend that reads the 
OpenTelemetry GenAI conventions (here a self-hosted Langfuse 4.81 receiving the 
spans of {{camel-opentelemetry2}} over OTLP). The route classifies a support 
question with {{openai}} and hands order questions to a {{langchain4j-agent}} 
that calls an {{ai-tool}} route ({{getOrderStatus}}). The model is a local 
Qwen3-4B, so no cost is shown.

*Before.* The agent run is one {{generate_content}} generation with 469 tokens, 
the sum of its two model calls. The tool is a plain {{ai-tool}} span, and no 
observation has an input or an output.
!1-before-camel-agent-run.png|width=900!

*Before, with {{camel-observability-services}} and the OpenTelemetry Java 
agent.* Each {{openai}} call is recorded twice, once by Camel (under the 
{{openai}} span) and once by the agent's {{openai-java}} instrumentation (its 
sibling), so its tokens are counted twice.
!2-before-otel-agent-duplicates.png|width=900!

*After, {{camel.aiObservability.level=full}}.* The agent run is an 
{{invoke_agent}} span with one generation per model call. The first one shows 
the user message and the {{getOrderStatus}} tool call that the model asked for.
!3-after-full-model-call.png|width=900!

*After.* The tool call is a tool observation, with its arguments and its result.
!4-after-full-tool-call.png|width=900!

*After, Quarkus with a quarkus-langchain4j model, {{level=detailed}}.* 
quarkus-langchain4j already traces the model calls ({{completion ...}}), so 
Camel does not record them again. Camel's agent run and route tool hold them in 
one tree.
!6-after-quarkus-langchain4j-deferred.png|width=900!

In the Camel Main "after" traces, the trace-level session and user come from 
the demo route, not from this change.

The camel-spring-boot and camel-quarkus parts are tracked separately.

_Claude Code on behalf of Croway_


  was:
Today the AI producers record one span per LLM call with the model, the token 
usage and the finish reason, the stable subset chosen in CAMEL-23861. Tracing 
backends built around LLM traces, which read the OpenTelemetry GenAI semantic 
conventions (for example Langfuse), need more to show what an AI route did, and 
several common setups record each call twice.

Tested end to end with a self-hosted Langfuse on Camel Main, Spring Boot 4.1 
and Quarkus 4, with {{openai}}, {{langchain4j-agent}} and an {{ai-tool}} route, 
{{spring-ai-chat}} and {{langchain4j-chat}}:
* no prompt or completion is recorded on any LLM call;
* a {{langchain4j-agent}} run is one {{generate_content}} span that sums every 
model call of the tool loop, and the tool calls are not traced as tools;
* the same call is recorded twice, with its tokens, by the OpenTelemetry Java 
agent (its {{openai-java}} instrumentation), by Spring AI ({{ChatModel}} 
observations) and by quarkus-langchain4j ({{SpanChatModelListener}});
* with both {{camel-opentelemetry2}} and a Micrometer {{ObservationRegistry}} 
(Spring Boot), the Micrometer path is taken: it drops the token usage, the 
response model and the finish reason, and records nothing when no tracing 
handler is registered;
* {{camel-observability-services}} does not include {{camel-ai-observability}}.

Proposal, keeping today's output as the default:
* {{camel.aiObservability.level}} = {{basic}} (default, unchanged) | 
{{detailed}} | {{full}}
** {{detailed}}: an {{invoke_agent}} span per {{langchain4j-agent}} run and per 
{{openai}} tool loop, one {{chat}} span per model call, an {{execute_tool}} 
span per tool call (Camel route tools through {{AiToolExecutor}}, MCP tools of 
{{openai}}), request parameters, cached tokens, {{gen_ai.provider.name}} and 
{{gen_ai.conversation.id}};
** {{full}}: {{gen_ai.input.messages}}, {{gen_ai.output.messages}}, 
{{gen_ai.system_instructions}}, tool arguments and results, each text cut at 
{{camel.aiObservability.contentMaxLength}} (8192).
* {{camel.aiObservability.deferToFramework}} (default true): no Camel span for 
the calls that Spring AI, quarkus-langchain4j, a {{langchain4j-observation}} 
listener or the OpenTelemetry Java agent already trace.
* Metrics are the same at every level. Agent runs and tool calls recorded 
through Micrometer are named {{camel.ai.agent}} and {{camel.ai.tool}}, apart 
from {{gen_ai.client.operation}}.
* At every level: numeric token attributes, the usage as high-cardinality 
values of the Micrometer Observation (as Spring AI does), GenAI spans through 
{{camel-opentelemetry2}} when it is in use, and GenAI spans made current during 
synchronous calls so that the spans of the HTTP client or the AI framework nest 
under them.
* {{camel-observability-services}} depends on {{camel-ai-observability}}.

The camel-spring-boot and camel-quarkus parts are tracked separately.

_Claude Code on behalf of Croway_



> camel-ai-observability - observability levels with agent, tool and message 
> data, and no double recording of calls traced by AI frameworks
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27131
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27131
>             Project: Camel
>          Issue Type: Sub-task
>          Components: camel-ai, camel-langchain4j-agent, camel-openai, 
> camel-spring-ai, camel-telemetry
>            Reporter: Federico Mariani
>            Assignee: Federico Mariani
>            Priority: Major
>         Attachments: 1-before-camel-agent-run.png, 
> 2-before-otel-agent-duplicates.png, 3-after-full-model-call.png, 
> 4-after-full-tool-call.png, 6-after-quarkus-langchain4j-deferred.png
>
>
> Today the AI producers record one span per LLM call with the model, the token 
> usage and the finish reason, the stable subset chosen in CAMEL-23861. Tracing 
> backends built around LLM traces, which read the OpenTelemetry GenAI semantic 
> conventions (for example Langfuse), need more to show what an AI route did, 
> and several common setups record each call twice.
> Tested end to end with a self-hosted Langfuse on Camel Main, Spring Boot 4.1 
> and Quarkus 4, with {{openai}}, {{langchain4j-agent}} and an {{ai-tool}} 
> route, {{spring-ai-chat}} and {{langchain4j-chat}}:
> * no prompt or completion is recorded on any LLM call;
> * a {{langchain4j-agent}} run is one {{generate_content}} span that sums 
> every model call of the tool loop, and the tool calls are not traced as tools;
> * the same call is recorded twice, with its tokens, by the OpenTelemetry Java 
> agent (its {{openai-java}} instrumentation), by Spring AI ({{ChatModel}} 
> observations) and by quarkus-langchain4j ({{SpanChatModelListener}});
> * with both {{camel-opentelemetry2}} and a Micrometer {{ObservationRegistry}} 
> (Spring Boot), the Micrometer path is taken: it drops the token usage, the 
> response model and the finish reason, and records nothing when no tracing 
> handler is registered;
> * {{camel-observability-services}} does not include 
> {{camel-ai-observability}}.
> Proposal, keeping today's output as the default:
> * {{camel.aiObservability.level}} = {{basic}} (default, unchanged) | 
> {{detailed}} | {{full}}
> ** {{detailed}}: an {{invoke_agent}} span per {{langchain4j-agent}} run and 
> per {{openai}} tool loop, one {{chat}} span per model call, an 
> {{execute_tool}} span per tool call (Camel route tools through 
> {{AiToolExecutor}}, MCP tools of {{openai}}), request parameters, cached 
> tokens, {{gen_ai.provider.name}} and {{gen_ai.conversation.id}};
> ** {{full}}: {{gen_ai.input.messages}}, {{gen_ai.output.messages}}, 
> {{gen_ai.system_instructions}}, tool arguments and results, each text cut at 
> {{camel.aiObservability.contentMaxLength}} (8192).
> * {{camel.aiObservability.deferToFramework}} (default true): no Camel span 
> for the calls that Spring AI, quarkus-langchain4j, a 
> {{langchain4j-observation}} listener or the OpenTelemetry Java agent already 
> trace.
> * Metrics are the same at every level. Agent runs and tool calls recorded 
> through Micrometer are named {{camel.ai.agent}} and {{camel.ai.tool}}, apart 
> from {{gen_ai.client.operation}}.
> * At every level: numeric token attributes, the usage as high-cardinality 
> values of the Micrometer Observation (as Spring AI does), GenAI spans through 
> {{camel-opentelemetry2}} when it is in use, and GenAI spans made current 
> during synchronous calls so that the spans of the HTTP client or the AI 
> framework nest under them.
> * {{camel-observability-services}} depends on {{camel-ai-observability}}.
> h3. Before and after
> What a Camel AI route looks like in a tracing backend that reads the 
> OpenTelemetry GenAI conventions (here a self-hosted Langfuse 4.81 receiving 
> the spans of {{camel-opentelemetry2}} over OTLP). The route classifies a 
> support question with {{openai}} and hands order questions to a 
> {{langchain4j-agent}} that calls an {{ai-tool}} route ({{getOrderStatus}}). 
> The model is a local Qwen3-4B, so no cost is shown.
> *Before.* The agent run is one {{generate_content}} generation with 469 
> tokens, the sum of its two model calls. The tool is a plain {{ai-tool}} span, 
> and no observation has an input or an output.
> !1-before-camel-agent-run.png|width=900!
> *Before, with {{camel-observability-services}} and the OpenTelemetry Java 
> agent.* Each {{openai}} call is recorded twice, once by Camel (under the 
> {{openai}} span) and once by the agent's {{openai-java}} instrumentation (its 
> sibling), so its tokens are counted twice.
> !2-before-otel-agent-duplicates.png|width=900!
> *After, {{camel.aiObservability.level=full}}.* The agent run is an 
> {{invoke_agent}} span with one generation per model call. The first one shows 
> the user message and the {{getOrderStatus}} tool call that the model asked 
> for.
> !3-after-full-model-call.png|width=900!
> *After.* The tool call is a tool observation, with its arguments and its 
> result.
> !4-after-full-tool-call.png|width=900!
> *After, Quarkus with a quarkus-langchain4j model, {{level=detailed}}.* 
> quarkus-langchain4j already traces the model calls ({{completion ...}}), so 
> Camel does not record them again. Camel's agent run and route tool hold them 
> in one tree.
> !6-after-quarkus-langchain4j-deferred.png|width=900!
> In the Camel Main "after" traces, the trace-level session and user come from 
> the demo route, not from this change.
> The camel-spring-boot and camel-quarkus parts are tracked separately.
> _Claude Code on behalf of Croway_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to