Federico Mariani created CAMEL-27131:
----------------------------------------

             Summary: camel-ai-observability - observability levels with agent, 
tool and message data, and no double recording of calls traced by AI frameworks
                 Key: CAMEL-27131
                 URL: https://issues.apache.org/jira/browse/CAMEL-27131
             Project: Camel
          Issue Type: Sub-task
          Components: camel-spring-ai, camel-langchain4j-agent, 
camel-telemetry, camel-ai, camel-openai
            Reporter: Federico Mariani


Today the AI producers record one span per LLM call with the model, the token 
usage and the finish reason, the stable subset chosen in CAMEL-23861. Tracing 
backends built around LLM traces, which read the OpenTelemetry GenAI semantic 
conventions (for example Langfuse), need more to show what an AI route did, and 
several common setups record each call twice.

Tested end to end with a self-hosted Langfuse on Camel Main, Spring Boot 4.1 
and Quarkus 4, with {{openai}}, {{langchain4j-agent}} and an {{ai-tool}} route, 
{{spring-ai-chat}} and {{langchain4j-chat}}:
* no prompt or completion is recorded on any LLM call;
* a {{langchain4j-agent}} run is one {{generate_content}} span that sums every 
model call of the tool loop, and the tool calls are not traced as tools;
* the same call is recorded twice, with its tokens, by the OpenTelemetry Java 
agent (its {{openai-java}} instrumentation), by Spring AI ({{ChatModel}} 
observations) and by quarkus-langchain4j ({{SpanChatModelListener}});
* with both {{camel-opentelemetry2}} and a Micrometer {{ObservationRegistry}} 
(Spring Boot), the Micrometer path is taken: it drops the token usage, the 
response model and the finish reason, and records nothing when no tracing 
handler is registered;
* {{camel-observability-services}} does not include {{camel-ai-observability}}.

Proposal, keeping today's output as the default:
* {{camel.aiObservability.level}} = {{basic}} (default, unchanged) | 
{{detailed}} | {{full}}
** {{detailed}}: an {{invoke_agent}} span per {{langchain4j-agent}} run and per 
{{openai}} tool loop, one {{chat}} span per model call, an {{execute_tool}} 
span per tool call (Camel route tools through {{AiToolExecutor}}, MCP tools of 
{{openai}}), request parameters, cached tokens, {{gen_ai.provider.name}} and 
{{gen_ai.conversation.id}};
** {{full}}: {{gen_ai.input.messages}}, {{gen_ai.output.messages}}, 
{{gen_ai.system_instructions}}, tool arguments and results, each text cut at 
{{camel.aiObservability.contentMaxLength}} (8192).
* {{camel.aiObservability.deferToFramework}} (default true): no Camel span for 
the calls that Spring AI, quarkus-langchain4j, a {{langchain4j-observation}} 
listener or the OpenTelemetry Java agent already trace.
* Metrics are the same at every level. Agent runs and tool calls recorded 
through Micrometer are named {{camel.ai.agent}} and {{camel.ai.tool}}, apart 
from {{gen_ai.client.operation}}.
* At every level: numeric token attributes, the usage as high-cardinality 
values of the Micrometer Observation (as Spring AI does), GenAI spans through 
{{camel-opentelemetry2}} when it is in use, and GenAI spans made current during 
synchronous calls so that the spans of the HTTP client or the AI framework nest 
under them.
* {{camel-observability-services}} depends on {{camel-ai-observability}}.

The camel-spring-boot and camel-quarkus parts are tracked separately.

_Claude Code on behalf of Croway_




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to