Claus Ibsen created CAMEL-27135:
-----------------------------------
Summary: camel-jbang - camel monitor shows the agent runs, tool
calls and messages of the detailed and full GenAI observability levels
Key: CAMEL-27135
URL: https://issues.apache.org/jira/browse/CAMEL-27135
Project: Camel
Issue Type: Sub-task
Components: camel-jbang
Reporter: Claus Ibsen
Fix For: 4.24.0
CAMEL-27131 (PR https://github.com/apache/camel/pull/30291) adds the
observability levels {{camel.aiObservability.level}} = {{basic}} (default) |
{{detailed}} | {{full}}. At {{detailed}}, an agent run is an {{invoke_agent}}
span with one {{chat}} span per model call and one {{execute_tool}} span per
tool call, plus cached tokens, {{gen_ai.provider.name}} and
{{gen_ai.conversation.id}}. At {{full}}, the spans also carry
{{gen_ai.input.messages}}, {{gen_ai.output.messages}},
{{gen_ai.system_instructions}} and the tool arguments and results.
camel-jbang's monitor already reads the GenAI spans from the span dump of the
running integration: the AI usage view and the Ollama view through
{{GenAiSpanUsageExtractor}}, and the Spans view (tree plus every attribute).
This ticket makes it use the new data.
*1. Count only model calls (small fix, needed once CAMEL-27131 is merged)*
{{GenAiSpanUsageExtractor.isGenAiSpan}} takes every span with
{{gen_ai.operation.name}} as one model call. At {{detailed}} the
{{invoke_agent}} and {{execute_tool}} spans would be listed as calls with 0
tokens, and with the agent or tool name in place of the model. The token totals
stay right, because CAMEL-27131 keeps the usage off the agent span. Count only
the model operations (chat, generate_content, embeddings, ...) in the usage and
Ollama views.
*2. Show the messages as a conversation*
At {{full}}, the messages are JSON strings of up to 8192 characters per text
(the default {{contentMaxLength}}), shown as one line in the span details.
Render them as a transcript: the system instructions, the user and assistant
messages, and each tool call with its arguments and result.
*3. Agent runs*
Group a run ({{invoke_agent}}) with its model calls and tool calls, with the
tokens and the time of each step, the cached tokens and the conversation id.
Make it clear when a level below {{detailed}} is in use, so nothing seems
missing.
*Also*
* Calls traced by an AI framework ({{camel.aiObservability.deferToFramework}},
default true) get no Camel span. That does not apply to a plain {{camel run}},
but the views should not assume a Camel GenAI span for every call.
* The MCP server tools that report AI activity could add the agent-run steps
from the same spans.
The work goes on main only, after CAMEL-27131 is merged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)