Federico Mariani created CAMEL-27131:
----------------------------------------
Summary: camel-ai-observability - observability levels with agent,
tool and message data, and no double recording of calls traced by AI frameworks
Key: CAMEL-27131
URL: https://issues.apache.org/jira/browse/CAMEL-27131
Project: Camel
Issue Type: Sub-task
Components: camel-spring-ai, camel-langchain4j-agent,
camel-telemetry, camel-ai, camel-openai
Reporter: Federico Mariani
Today the AI producers record one span per LLM call with the model, the token
usage and the finish reason, the stable subset chosen in CAMEL-23861. Tracing
backends built around LLM traces, which read the OpenTelemetry GenAI semantic
conventions (for example Langfuse), need more to show what an AI route did, and
several common setups record each call twice.
Tested end to end with a self-hosted Langfuse on Camel Main, Spring Boot 4.1
and Quarkus 4, with {{openai}}, {{langchain4j-agent}} and an {{ai-tool}} route,
{{spring-ai-chat}} and {{langchain4j-chat}}:
* no prompt or completion is recorded on any LLM call;
* a {{langchain4j-agent}} run is one {{generate_content}} span that sums every
model call of the tool loop, and the tool calls are not traced as tools;
* the same call is recorded twice, with its tokens, by the OpenTelemetry Java
agent (its {{openai-java}} instrumentation), by Spring AI ({{ChatModel}}
observations) and by quarkus-langchain4j ({{SpanChatModelListener}});
* with both {{camel-opentelemetry2}} and a Micrometer {{ObservationRegistry}}
(Spring Boot), the Micrometer path is taken: it drops the token usage, the
response model and the finish reason, and records nothing when no tracing
handler is registered;
* {{camel-observability-services}} does not include {{camel-ai-observability}}.
Proposal, keeping today's output as the default:
* {{camel.aiObservability.level}} = {{basic}} (default, unchanged) |
{{detailed}} | {{full}}
** {{detailed}}: an {{invoke_agent}} span per {{langchain4j-agent}} run and per
{{openai}} tool loop, one {{chat}} span per model call, an {{execute_tool}}
span per tool call (Camel route tools through {{AiToolExecutor}}, MCP tools of
{{openai}}), request parameters, cached tokens, {{gen_ai.provider.name}} and
{{gen_ai.conversation.id}};
** {{full}}: {{gen_ai.input.messages}}, {{gen_ai.output.messages}},
{{gen_ai.system_instructions}}, tool arguments and results, each text cut at
{{camel.aiObservability.contentMaxLength}} (8192).
* {{camel.aiObservability.deferToFramework}} (default true): no Camel span for
the calls that Spring AI, quarkus-langchain4j, a {{langchain4j-observation}}
listener or the OpenTelemetry Java agent already trace.
* Metrics are the same at every level. Agent runs and tool calls recorded
through Micrometer are named {{camel.ai.agent}} and {{camel.ai.tool}}, apart
from {{gen_ai.client.operation}}.
* At every level: numeric token attributes, the usage as high-cardinality
values of the Micrometer Observation (as Spring AI does), GenAI spans through
{{camel-opentelemetry2}} when it is in use, and GenAI spans made current during
synchronous calls so that the spans of the HTTP client or the AI framework nest
under them.
* {{camel-observability-services}} depends on {{camel-ai-observability}}.
The camel-spring-boot and camel-quarkus parts are tracked separately.
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)