luigidemasi opened a new issue, #50: URL: https://github.com/apache/camel-kit/issues/50
> Imported from [luigidemasi/camel-kit#88](https://github.com/luigidemasi/camel-kit/issues/88), opened by [luigidemasi](https://github.com/luigidemasi) on 2026-05-18. --- ## Description Add a performance testing framework for bottleneck optimization across the full Camel Kit stack: LLM interaction layer (token consumption, context buildup, skill execution latency) and Java runtime layer (MCP server, Lucene search, ONNX embedding). **ADR:** `docs/adr/0034-performance-testing-otel-framework.md` ## Approach: Unified OTel Trace Pipeline Three instrumentation layers feed a single Honeycomb backend via W3C TRACEPARENT propagation: 1. **Java test harness** (JUnit 5 + `ProcessBuilder` + `claude -p`) — scripted scenarios with canned prompts for reproducibility. Pure Java, no new language dependencies. `ProcessBuilder` provides the same OTel capabilities as the Python/TypeScript Agent SDK since the telemetry lives in the CLI itself. 2. **Claude Code native OTel** — automatic spans for interactions, LLM requests, tool calls; token counters (input/output/cache_read/cache_creation) 3. **Quarkus OTel** (Knowledge MCP server) — `@WithSpan` on IndexResolver, embedding, and hybrid search ## Key Measurement Dimensions - Wall-clock latency per tool call, LLM request, and scenario - Token consumption (input, output, cache read, cache creation) per API call and per skill - Context buildup: growth rate, peak size, cache hit ratio, compression events - MCP tool latency ranking - ONNX embedding vs Lucene search split ## Implementation Steps 1. Create `camel-kit-perf/` Maven module with JUnit 5 dependency 2. Implement `ClaudePerfRunner` (ProcessBuilder wrapper for `claude -p --bare --output-format json`) 3. Implement `AbstractPerfScenario` with OTel env var configuration 4. Write first scenario: `CamelKnowledgeScenarioTest` (simplest — single MCP call) 5. Add `quarkus-opentelemetry` to Knowledge MCP server `pom.xml` 6. Add `@WithSpan` to IndexResolver, embedding, and search methods 7. Configure `application.properties` for OTel (disabled by default, enabled via env vars) 8. Run first end-to-end trace and verify it appears in Honeycomb 9. Write remaining scenarios (brainstorm, migrate, ship) 10. Run analysis queries in Honeycomb to identify bottlenecks ## Additional Context - Tracing is toggled on/off via environment variables — zero overhead in normal usage - `--bare` mode ensures reproducible runs (skips hooks, plugins, auto-discovery) - `--output-format json` returns session ID, cost, and token usage in the response - Claude Code first, extensible to other agents (Bob, Gemini, Qwen, OpenCode) later - Camel MCP server (upstream) is not instrumented — Claude Code's OTel gives sufficient visibility - Python Agent SDK was evaluated and rejected — it's a thin wrapper around `claude -p` that adds no unique OTel capability, while introducing a second language ecosystem - Anthropic Managed Agents Java SDK was evaluated and rejected — it runs in Anthropic's cloud and cannot access local MCP servers or skill files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
