davsclaus commented on code in PR #1734: URL: https://github.com/apache/camel-website/pull/1734#discussion_r3888839793
########## content/blog/2026/08/camel-genai-observability-jbang/index.md: ########## @@ -0,0 +1,222 @@ +--- +title: "Observe Your Camel AI Routes with GenAI OpenTelemetry" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "opentelemetry", "ollama", "langchain4j", "camel jbang", "camel tui", "camel 4.23", "llm"] +preview: "Apache Camel 4.23 adds GenAI observability — OpenTelemetry spans and Micrometer metrics for LLM routes. Prototype it in minutes with JBang, Ollama, and the Camel TUI." +--- + +Once LLMs live inside Camel routes, the next question is always the same: *how much are we spending, and +where is latency coming from?* Apache Camel **4.23** introduces **GenAI observability** — OpenTelemetry spans +and Micrometer metrics for LLM producers, aligned with the +[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). + +This is **Blog 1** in a two-part series. We start with the fastest path to visible AI telemetry: Camel JBang, Review Comment: Camel CLI We use the term CLI over jbang as CLI is much better understood also by AIs ########## content/blog/2026/08/camel-genai-observability-jbang/index.md: ########## @@ -0,0 +1,222 @@ +--- +title: "Observe Your Camel AI Routes with GenAI OpenTelemetry" +date: 2026-08-29 Review Comment: Change date when its to be published ########## content/blog/2026/08/camel-genai-observability-jbang/index.md: ########## @@ -0,0 +1,222 @@ +--- +title: "Observe Your Camel AI Routes with GenAI OpenTelemetry" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "opentelemetry", "ollama", "langchain4j", "camel jbang", "camel tui", "camel 4.23", "llm"] +preview: "Apache Camel 4.23 adds GenAI observability — OpenTelemetry spans and Micrometer metrics for LLM routes. Prototype it in minutes with JBang, Ollama, and the Camel TUI." +--- + +Once LLMs live inside Camel routes, the next question is always the same: *how much are we spending, and +where is latency coming from?* Apache Camel **4.23** introduces **GenAI observability** — OpenTelemetry spans +and Micrometer metrics for LLM producers, aligned with the +[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). + +This is **Blog 1** in a two-part series. We start with the fastest path to visible AI telemetry: Camel JBang, +a LangChain4j chat route, [Ollama](https://ollama.com/), and the Camel CLI/TUI — no Spring Boot required. + +Special thanks to **Claus Ibsen** for guidance on shaping this material. + +## The three phases of Camel Gen AI + +| Phase | Name | What you do | +|-------|------|-------------| +| 1 | **Connect** | Wire LLMs into routes (`langchain4j-chat`, `openai`, Spring AI) | +| 2 | **Act** | Give AI tools via `ai-tool:`, agents, MCP server, A2A | +| 3 | **Operate** | Run in production with guardrails, RAG, and **GenAI observability** | + +This post covers Phase 1 plus the first observability prototype. +[Part 2](/blog/2026/08/camel-genai-observability-spring-boot/) moves to Spring Boot and a full observability +stack (Prometheus, VictoriaTraces, Perses). + +## What you'll build + +A timer-driven route that calls Ollama every 15 seconds. For each LLM invocation you get: + +- Micrometer timer: `gen_ai.client.operation` +- Micrometer counter: `gen_ai.client.token.usage` (tags `input` / `output`) +- OpenTelemetry child spans with `gen_ai.request.model`, token counts, finish reason +- Exchange headers: `CamelLangChain4jChatRequestModel`, `CamelLangChain4jChatResponseModel` +- TUI **Spans** tab visualization + AI Usage view (**Ctrl+U**) + +``` +timer:genai (every 15s) + └─ langchain4j-chat (Ollama ChatModel) + ├─ camel-ai-observability-api → start GenAI span + record metrics + ├─ camel-opentelemetry2 → export span attributes + └─ camel-micrometer → gen_ai.client.* metrics + +camel run --observe + └─ camel-observability-services + ├─ /observe/health + ├─ /observe/metrics (Prometheus format) + └─ TUI Spans collector (embedded OTLP for dev) + +camel tui + ├─ Spans tab (shortcut: o) + └─ AI panel + Ctrl+U (CLI ask + route GenAI usage combined) +``` + +## Prerequisites + +- Camel JBang **4.23+** (`camel version`) +- Ollama installed and running + +```shell +ollama pull llama3.2 +ollama serve +``` + +## Run the example + +The bundled example ships with Camel 4.23 at +`dsl/camel-jbang/camel-jbang-core/src/main/resources/examples/ai/genai-observability/` +([source on GitHub](https://github.com/apache/camel/pull/25891/files)). + +```shell +cd ai/genai-observability + +camel run GenAiObservabilityRoute.java application.properties \ + --observe \ + --dependency=camel-langchain4j-chat \ + --dependency=camel-ai-observability \ + --dependency=langchain4j-ollama +``` + +The `--observe` flag adds `camel-observability-services`, enables health checks, Micrometer metrics, +OpenTelemetry tracing, and powers the TUI **Spans** tab on the management port (default `9876`). + +Wait for log lines like: + +```text +LLM reply: Apache Camel is an open source integration framework... +Request model: llama3.2 +Response model: llama3.2 +``` + +## Inspect Prometheus metrics + +```shell +curl -s http://127.0.0.1:9876/observe/metrics | grep gen_ai +``` + +Example output (abbreviated): + +```text +# HELP gen_ai_client_operation GenAI client operation duration +gen_ai_client_operation_count{gen_ai_operation_name="chat",gen_ai_system="langchain4j",...} 3.0 + +# HELP gen_ai_client_token_usage GenAI token usage +gen_ai_client_token_usage_total{gen_ai_token_type="input",...} 42.0 +gen_ai_client_token_usage_total{gen_ai_token_type="output",...} 18.0 +``` + +## Explore spans in the TUI + +Open a second terminal: + +```shell +camel tui +``` + +1. Select the running integration (`genai-observability` or similar) +2. Press **o** or navigate to **More → Spans** +3. Wait for the next timer tick (~15s) and watch a new span appear + +Key span attributes: + +| Attribute | Meaning | +|-----------|---------| +| `gen_ai.operation.name` | e.g. `chat`, `embeddings` | +| `gen_ai.system` | Provider abstraction (e.g. `langchain4j`) | +| `gen_ai.request.model` | Model requested | +| `gen_ai.response.model` | Model that served the response | +| `gen_ai.usage.input_tokens` | Prompt tokens | +| `gen_ai.usage.output_tokens` | Completion tokens | +| `camel.component` | e.g. `langchain4j-chat` | + +Press **Ctrl+U** in the AI panel to toggle the **AI Usage** view — token consumption from both +TUI `camel ask` prompts and route LLM calls on one screen. + +## Example route + +```java +import dev.langchain4j.model.chat.ChatModel; +import dev.langchain4j.model.ollama.OllamaChatModel; +import org.apache.camel.builder.RouteBuilder; + +import static java.time.Duration.ofSeconds; + +public class GenAiObservabilityRoute extends RouteBuilder { + + @Override + public void configure() { + ChatModel chatModel = OllamaChatModel.builder() Review Comment: Here is something to improve for the future (forage can do something for this I think) but in general for Camel CLI we should also make this possible, so you can do this easier in yaml-dsl as well. We have Camel on a mission to be useable for non Java coders as well. ########## content/blog/2026/08/camel-genai-observability-jbang/index.md: ########## @@ -0,0 +1,222 @@ +--- +title: "Observe Your Camel AI Routes with GenAI OpenTelemetry" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "opentelemetry", "ollama", "langchain4j", "camel jbang", "camel tui", "camel 4.23", "llm"] +preview: "Apache Camel 4.23 adds GenAI observability — OpenTelemetry spans and Micrometer metrics for LLM routes. Prototype it in minutes with JBang, Ollama, and the Camel TUI." +--- + +Once LLMs live inside Camel routes, the next question is always the same: *how much are we spending, and +where is latency coming from?* Apache Camel **4.23** introduces **GenAI observability** — OpenTelemetry spans +and Micrometer metrics for LLM producers, aligned with the +[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). + +This is **Blog 1** in a two-part series. We start with the fastest path to visible AI telemetry: Camel JBang, +a LangChain4j chat route, [Ollama](https://ollama.com/), and the Camel CLI/TUI — no Spring Boot required. + +Special thanks to **Claus Ibsen** for guidance on shaping this material. Review Comment: Oh that is not needed in the blog ########## content/blog/2026/08/camel-genai-observability-jbang/index.md: ########## @@ -0,0 +1,222 @@ +--- +title: "Observe Your Camel AI Routes with GenAI OpenTelemetry" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "opentelemetry", "ollama", "langchain4j", "camel jbang", "camel tui", "camel 4.23", "llm"] +preview: "Apache Camel 4.23 adds GenAI observability — OpenTelemetry spans and Micrometer metrics for LLM routes. Prototype it in minutes with JBang, Ollama, and the Camel TUI." +--- + +Once LLMs live inside Camel routes, the next question is always the same: *how much are we spending, and +where is latency coming from?* Apache Camel **4.23** introduces **GenAI observability** — OpenTelemetry spans +and Micrometer metrics for LLM producers, aligned with the +[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). + +This is **Blog 1** in a two-part series. We start with the fastest path to visible AI telemetry: Camel JBang, +a LangChain4j chat route, [Ollama](https://ollama.com/), and the Camel CLI/TUI — no Spring Boot required. + +Special thanks to **Claus Ibsen** for guidance on shaping this material. + +## The three phases of Camel Gen AI + +| Phase | Name | What you do | +|-------|------|-------------| +| 1 | **Connect** | Wire LLMs into routes (`langchain4j-chat`, `openai`, Spring AI) | +| 2 | **Act** | Give AI tools via `ai-tool:`, agents, MCP server, A2A | +| 3 | **Operate** | Run in production with guardrails, RAG, and **GenAI observability** | + +This post covers Phase 1 plus the first observability prototype. +[Part 2](/blog/2026/08/camel-genai-observability-spring-boot/) moves to Spring Boot and a full observability +stack (Prometheus, VictoriaTraces, Perses). + +## What you'll build + +A timer-driven route that calls Ollama every 15 seconds. For each LLM invocation you get: + +- Micrometer timer: `gen_ai.client.operation` +- Micrometer counter: `gen_ai.client.token.usage` (tags `input` / `output`) +- OpenTelemetry child spans with `gen_ai.request.model`, token counts, finish reason +- Exchange headers: `CamelLangChain4jChatRequestModel`, `CamelLangChain4jChatResponseModel` +- TUI **Spans** tab visualization + AI Usage view (**Ctrl+U**) + +``` +timer:genai (every 15s) + └─ langchain4j-chat (Ollama ChatModel) + ├─ camel-ai-observability-api → start GenAI span + record metrics + ├─ camel-opentelemetry2 → export span attributes + └─ camel-micrometer → gen_ai.client.* metrics + +camel run --observe + └─ camel-observability-services + ├─ /observe/health + ├─ /observe/metrics (Prometheus format) + └─ TUI Spans collector (embedded OTLP for dev) + +camel tui + ├─ Spans tab (shortcut: o) + └─ AI panel + Ctrl+U (CLI ask + route GenAI usage combined) +``` + +## Prerequisites + +- Camel JBang **4.23+** (`camel version`) +- Ollama installed and running + +```shell +ollama pull llama3.2 +ollama serve +``` + +## Run the example + +The bundled example ships with Camel 4.23 at Review Comment: The example should be in camel-jbang-examples, we can then mark it as bundled for TUI so you can launch it there directly without CLI commands (as well) ########## content/blog/2026/08/camel-genai-observability-jbang/index.md: ########## @@ -0,0 +1,222 @@ +--- +title: "Observe Your Camel AI Routes with GenAI OpenTelemetry" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "opentelemetry", "ollama", "langchain4j", "camel jbang", "camel tui", "camel 4.23", "llm"] +preview: "Apache Camel 4.23 adds GenAI observability — OpenTelemetry spans and Micrometer metrics for LLM routes. Prototype it in minutes with JBang, Ollama, and the Camel TUI." +--- + +Once LLMs live inside Camel routes, the next question is always the same: *how much are we spending, and +where is latency coming from?* Apache Camel **4.23** introduces **GenAI observability** — OpenTelemetry spans +and Micrometer metrics for LLM producers, aligned with the +[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). + +This is **Blog 1** in a two-part series. We start with the fastest path to visible AI telemetry: Camel JBang, +a LangChain4j chat route, [Ollama](https://ollama.com/), and the Camel CLI/TUI — no Spring Boot required. + +Special thanks to **Claus Ibsen** for guidance on shaping this material. + +## The three phases of Camel Gen AI + +| Phase | Name | What you do | +|-------|------|-------------| +| 1 | **Connect** | Wire LLMs into routes (`langchain4j-chat`, `openai`, Spring AI) | +| 2 | **Act** | Give AI tools via `ai-tool:`, agents, MCP server, A2A | +| 3 | **Operate** | Run in production with guardrails, RAG, and **GenAI observability** | + +This post covers Phase 1 plus the first observability prototype. +[Part 2](/blog/2026/08/camel-genai-observability-spring-boot/) moves to Spring Boot and a full observability +stack (Prometheus, VictoriaTraces, Perses). + +## What you'll build + +A timer-driven route that calls Ollama every 15 seconds. For each LLM invocation you get: + +- Micrometer timer: `gen_ai.client.operation` +- Micrometer counter: `gen_ai.client.token.usage` (tags `input` / `output`) +- OpenTelemetry child spans with `gen_ai.request.model`, token counts, finish reason +- Exchange headers: `CamelLangChain4jChatRequestModel`, `CamelLangChain4jChatResponseModel` +- TUI **Spans** tab visualization + AI Usage view (**Ctrl+U**) + +``` +timer:genai (every 15s) + └─ langchain4j-chat (Ollama ChatModel) + ├─ camel-ai-observability-api → start GenAI span + record metrics + ├─ camel-opentelemetry2 → export span attributes + └─ camel-micrometer → gen_ai.client.* metrics + +camel run --observe + └─ camel-observability-services + ├─ /observe/health + ├─ /observe/metrics (Prometheus format) + └─ TUI Spans collector (embedded OTLP for dev) + +camel tui + ├─ Spans tab (shortcut: o) + └─ AI panel + Ctrl+U (CLI ask + route GenAI usage combined) +``` + +## Prerequisites + +- Camel JBang **4.23+** (`camel version`) +- Ollama installed and running + +```shell +ollama pull llama3.2 +ollama serve +``` + +## Run the example + +The bundled example ships with Camel 4.23 at +`dsl/camel-jbang/camel-jbang-core/src/main/resources/examples/ai/genai-observability/` +([source on GitHub](https://github.com/apache/camel/pull/25891/files)). + +```shell +cd ai/genai-observability + +camel run GenAiObservabilityRoute.java application.properties \ + --observe \ + --dependency=camel-langchain4j-chat \ + --dependency=camel-ai-observability \ + --dependency=langchain4j-ollama +``` + +The `--observe` flag adds `camel-observability-services`, enables health checks, Micrometer metrics, +OpenTelemetry tracing, and powers the TUI **Spans** tab on the management port (default `9876`). + +Wait for log lines like: + +```text +LLM reply: Apache Camel is an open source integration framework... +Request model: llama3.2 +Response model: llama3.2 +``` + +## Inspect Prometheus metrics + +```shell +curl -s http://127.0.0.1:9876/observe/metrics | grep gen_ai +``` + +Example output (abbreviated): + +```text +# HELP gen_ai_client_operation GenAI client operation duration +gen_ai_client_operation_count{gen_ai_operation_name="chat",gen_ai_system="langchain4j",...} 3.0 + +# HELP gen_ai_client_token_usage GenAI token usage +gen_ai_client_token_usage_total{gen_ai_token_type="input",...} 42.0 +gen_ai_client_token_usage_total{gen_ai_token_type="output",...} 18.0 +``` + +## Explore spans in the TUI + +Open a second terminal: + +```shell +camel tui +``` + +1. Select the running integration (`genai-observability` or similar) +2. Press **o** or navigate to **More → Spans** +3. Wait for the next timer tick (~15s) and watch a new span appear + +Key span attributes: + +| Attribute | Meaning | +|-----------|---------| +| `gen_ai.operation.name` | e.g. `chat`, `embeddings` | +| `gen_ai.system` | Provider abstraction (e.g. `langchain4j`) | +| `gen_ai.request.model` | Model requested | +| `gen_ai.response.model` | Model that served the response | +| `gen_ai.usage.input_tokens` | Prompt tokens | +| `gen_ai.usage.output_tokens` | Completion tokens | +| `camel.component` | e.g. `langchain4j-chat` | + +Press **Ctrl+U** in the AI panel to toggle the **AI Usage** view — token consumption from both +TUI `camel ask` prompts and route LLM calls on one screen. + +## Example route + +```java +import dev.langchain4j.model.chat.ChatModel; +import dev.langchain4j.model.ollama.OllamaChatModel; +import org.apache.camel.builder.RouteBuilder; + +import static java.time.Duration.ofSeconds; + +public class GenAiObservabilityRoute extends RouteBuilder { + + @Override + public void configure() { + ChatModel chatModel = OllamaChatModel.builder() + .baseUrl("{{ollama.baseUrl:http://localhost:11434}}";) + .modelName("{{ollama.model:llama3.2}}") + .temperature(0.2) + .timeout(ofSeconds(120)) + .build(); + getContext().getRegistry().bind("chatModel", chatModel); + + from("timer:genai?period={{genai.period:15000}}") + .routeId("genai-chat") + .setBody(constant("In one sentence, what is Apache Camel integration?")) + .to("langchain4j-chat:demo?chatModel=#chatModel") + .log("LLM reply: ${body}") + .log("Request model: ${header.CamelLangChain4jChatRequestModel}") + .log("Response model: ${header.CamelLangChain4jChatResponseModel}"); + } +} +``` + +### application.properties + +```properties +ollama.baseUrl=http://localhost:11434 +ollama.model=llama3.2 +genai.period=15000 + +# GenAI observability — enabled by default when backends present +camel.ai.observability.enabled=true +``` + +GenAI observability also covers `openai:` — run with `--dependency=camel-openai` and the same +`--observe` flag; spans use the same `gen_ai.*` attributes. + +## What's instrumented today + +| Component | Operations observed | +|-----------|---------------------| +| `langchain4j-chat` | Chat completions | +| `langchain4j-tools` | Tool-augmented LLM calls | +| `langchain4j-agent` | AI Service agent loops | +| `langchain4j-embeddings` | Embedding generation | +| `openai` | Chat, embeddings, Responses API, etc. | + +## Troubleshooting + +| Symptom | Fix | +|---------|-----| +| No `gen_ai` metrics | Confirm `--observe` or manual observability config; check `camel-ai-observability` on classpath | +| Empty Spans tab | Wait for at least one LLM call; verify `camel.opentelemetry2.enabled=true` | +| Ollama connection refused | Run `ollama serve`; check `ollama.baseUrl` | +| Metrics port unreachable | Default management port is `9876`; look for `Management service available` in logs | + +## Next up — Spring Boot and the observability stack + +[Part 2](/blog/2026/08/camel-genai-observability-spring-boot/) wires the same `gen_ai.*` signals into Review Comment: Give a hine that you can use Camel CLI export command to turn this prototype into a spring boot based project where you can continue ########## content/blog/2026/08/camel-genai-observability-spring-boot/index.md: ########## @@ -0,0 +1,274 @@ +--- +title: "GenAI Observability with Spring Boot and the Camel Observability Stack" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "spring boot", "prometheus", "opentelemetry", "victoriatraces", "perses", "ollama", "langchain4j", "camel 4.23"] +preview: "Take Camel GenAI observability from JBang prototypes to production — Spring Boot Actuator, Prometheus scraping, OTLP traces in VictoriaTraces, and optional Perses dashboards." +--- + +In [Part 1](/blog/2026/08/camel-genai-observability-jbang/) we prototyped GenAI observability with Camel JBang +and the TUI. This follow-up — **Phase 3 (Operate)** — shows the same `gen_ai.*` telemetry in a **Spring Boot** +application wired to the observability stack Camel uses in test-infra: **Prometheus**, **VictoriaTraces**, and **Perses**. + +Special thanks to **Claus Ibsen** for guidance on this series. + +The runnable sample lives in +[`spring-boot/genai-observability`](https://github.com/apache/camel-examples/pull/255/files) on camel-examples +(open PR until merged to `main`). + +## Architecture + +``` +┌─────────────────┐ scrape ┌──────────────┐ Review Comment: Nice diagram ########## content/blog/2026/08/camel-genai-observability-spring-boot/index.md: ########## @@ -0,0 +1,274 @@ +--- +title: "GenAI Observability with Spring Boot and the Camel Observability Stack" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "spring boot", "prometheus", "opentelemetry", "victoriatraces", "perses", "ollama", "langchain4j", "camel 4.23"] +preview: "Take Camel GenAI observability from JBang prototypes to production — Spring Boot Actuator, Prometheus scraping, OTLP traces in VictoriaTraces, and optional Perses dashboards." +--- + +In [Part 1](/blog/2026/08/camel-genai-observability-jbang/) we prototyped GenAI observability with Camel JBang +and the TUI. This follow-up — **Phase 3 (Operate)** — shows the same `gen_ai.*` telemetry in a **Spring Boot** +application wired to the observability stack Camel uses in test-infra: **Prometheus**, **VictoriaTraces**, and **Perses**. + +Special thanks to **Claus Ibsen** for guidance on this series. + +The runnable sample lives in +[`spring-boot/genai-observability`](https://github.com/apache/camel-examples/pull/255/files) on camel-examples Review Comment: should be in camel-spring-boot-examples ########## content/blog/2026/08/camel-genai-observability-spring-boot/index.md: ########## @@ -0,0 +1,274 @@ +--- +title: "GenAI Observability with Spring Boot and the Camel Observability Stack" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "spring boot", "prometheus", "opentelemetry", "victoriatraces", "perses", "ollama", "langchain4j", "camel 4.23"] +preview: "Take Camel GenAI observability from JBang prototypes to production — Spring Boot Actuator, Prometheus scraping, OTLP traces in VictoriaTraces, and optional Perses dashboards." +--- + +In [Part 1](/blog/2026/08/camel-genai-observability-jbang/) we prototyped GenAI observability with Camel JBang +and the TUI. This follow-up — **Phase 3 (Operate)** — shows the same `gen_ai.*` telemetry in a **Spring Boot** +application wired to the observability stack Camel uses in test-infra: **Prometheus**, **VictoriaTraces**, and **Perses**. + +Special thanks to **Claus Ibsen** for guidance on this series. + +The runnable sample lives in +[`spring-boot/genai-observability`](https://github.com/apache/camel-examples/pull/255/files) on camel-examples +(open PR until merged to `main`). + +## Architecture + +``` +┌─────────────────┐ scrape ┌──────────────┐ +│ Spring Boot │ ──────────────► │ Prometheus │ +│ Camel + LLM │ │ :9090 │ +│ :8080 │ └──────┬───────┘ +└────────┬────────┘ │ + │ OTLP traces ▼ + ▼ ┌──────────────┐ +┌─────────────────┐ │ Perses │ +│ VictoriaTraces │ ◄── dashboards │ :8088 │ +│ :9428 │ └──────────────┘ +└─────────────────┘ + +Optional: camel tui ──► Spring Boot via cli-connector +``` + +For every LLM call in the YAML route: + +- Actuator `/actuator/prometheus` exposes `gen_ai_client_*` metrics +- OTLP export sends spans with token attributes to VictoriaTraces +- You query traces by `gen_ai.operation.name="chat"` + +## Quick start + +### Prerequisites + +```shell +java -version # 17+ +mvn -version # 3.9+ +docker --version # for observability stack +ollama pull llama3.2 +ollama serve +``` + +### Start the observability stack + +From the example directory: + +```shell +cd spring-boot/genai-observability +docker compose up -d +``` + +| Service | Port | Role | +|---------|------|------| +| Prometheus | 9090 | Scrapes Spring Boot Actuator Prometheus endpoint | +| VictoriaTraces | 9428 | Stores OTLP traces; UI at `/select/vmui` | +| Perses | 8088 | Metrics dashboards (optional visualization) | + +Container images match `camel-test-infra-observability` so CI and blog readers use the same stack. + +### Run Spring Boot + +```shell +mvn spring-boot:run +``` + +Wait for: + +```text +Started GenAiObservabilityApplication +LLM reply: Apache Camel is an integration framework... +Models: req=llama3.2 resp=llama3.2 +``` + +## Verify GenAI metrics + +```shell +curl -s http://localhost:8080/actuator/prometheus | grep gen_ai +``` + +Expected Micrometer names: + +- `gen_ai_client_operation` — timer/summary of LLM call duration +- `gen_ai_client_token_usage` — counter with tag `gen_ai_token_type=input|output` + +### Prometheus queries + +Open [http://localhost:9090](http://localhost:9090) and try: + Review Comment: Oh great with those queries, thanks ########## content/blog/2026/08/camel-genai-observability-spring-boot/index.md: ########## @@ -0,0 +1,274 @@ +--- +title: "GenAI Observability with Spring Boot and the Camel Observability Stack" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "spring boot", "prometheus", "opentelemetry", "victoriatraces", "perses", "ollama", "langchain4j", "camel 4.23"] +preview: "Take Camel GenAI observability from JBang prototypes to production — Spring Boot Actuator, Prometheus scraping, OTLP traces in VictoriaTraces, and optional Perses dashboards." +--- + +In [Part 1](/blog/2026/08/camel-genai-observability-jbang/) we prototyped GenAI observability with Camel JBang +and the TUI. This follow-up — **Phase 3 (Operate)** — shows the same `gen_ai.*` telemetry in a **Spring Boot** +application wired to the observability stack Camel uses in test-infra: **Prometheus**, **VictoriaTraces**, and **Perses**. + +Special thanks to **Claus Ibsen** for guidance on this series. + +The runnable sample lives in +[`spring-boot/genai-observability`](https://github.com/apache/camel-examples/pull/255/files) on camel-examples +(open PR until merged to `main`). + +## Architecture + +``` +┌─────────────────┐ scrape ┌──────────────┐ +│ Spring Boot │ ──────────────► │ Prometheus │ +│ Camel + LLM │ │ :9090 │ +│ :8080 │ └──────┬───────┘ +└────────┬────────┘ │ + │ OTLP traces ▼ + ▼ ┌──────────────┐ +┌─────────────────┐ │ Perses │ +│ VictoriaTraces │ ◄── dashboards │ :8088 │ +│ :9428 │ └──────────────┘ +└─────────────────┘ + +Optional: camel tui ──► Spring Boot via cli-connector +``` + +For every LLM call in the YAML route: + +- Actuator `/actuator/prometheus` exposes `gen_ai_client_*` metrics +- OTLP export sends spans with token attributes to VictoriaTraces +- You query traces by `gen_ai.operation.name="chat"` + +## Quick start + +### Prerequisites + +```shell +java -version # 17+ +mvn -version # 3.9+ +docker --version # for observability stack +ollama pull llama3.2 +ollama serve +``` + +### Start the observability stack + +From the example directory: + +```shell +cd spring-boot/genai-observability +docker compose up -d +``` + +| Service | Port | Role | +|---------|------|------| +| Prometheus | 9090 | Scrapes Spring Boot Actuator Prometheus endpoint | +| VictoriaTraces | 9428 | Stores OTLP traces; UI at `/select/vmui` | +| Perses | 8088 | Metrics dashboards (optional visualization) | + +Container images match `camel-test-infra-observability` so CI and blog readers use the same stack. + +### Run Spring Boot + +```shell +mvn spring-boot:run +``` + +Wait for: + +```text +Started GenAiObservabilityApplication +LLM reply: Apache Camel is an integration framework... +Models: req=llama3.2 resp=llama3.2 +``` + +## Verify GenAI metrics + +```shell +curl -s http://localhost:8080/actuator/prometheus | grep gen_ai +``` + +Expected Micrometer names: + +- `gen_ai_client_operation` — timer/summary of LLM call duration +- `gen_ai_client_token_usage` — counter with tag `gen_ai_token_type=input|output` + +### Prometheus queries + +Open [http://localhost:9090](http://localhost:9090) and try: + +```promql +# Total output tokens over time +increase(gen_ai_client_token_usage_total{gen_ai_token_type="output"}[1h]) + +# LLM call rate +rate(gen_ai_client_operation_count[5m]) + +# By Camel component +sum by (camel_component) (gen_ai_client_operation_count) +``` + +### Example alert rule (sketch) + +```yaml +groups: + - name: camel-genai + rules: + - alert: HighGenAITokenBurn + expr: rate(gen_ai_client_token_usage_total[15m]) > 1000 + for: 10m + labels: + severity: warning + annotations: + summary: "High GenAI token usage on {{ $labels.job }}" +``` + +## Explore traces in VictoriaTraces + +Open [http://localhost:9428/select/vmui](http://localhost:9428/select/vmui) + +Useful trace search filters: + +- `gen_ai.operation.name="chat"` +- `gen_ai.request.model="llama3.2"` +- `camel.component="langchain4j-chat"` + +Each span carries `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, +`gen_ai.response.finish_reasons`, and `gen_ai.system` (e.g. `langchain4j`). + +Correlate traces with logs by passing the trace ID from MDC — see the +[OpenTelemetry docs](/manual/camel-jbang-tui.html#_opentelemetry_spans). + +## Key configuration + +### Maven dependencies (excerpt) + +```xml +<dependency> + <groupId>org.apache.camel.springboot</groupId> + <artifactId>camel-spring-boot-starter</artifactId> +</dependency> +<dependency> + <groupId>org.apache.camel.springboot</groupId> + <artifactId>camel-observability-services-starter</artifactId> +</dependency> +<dependency> + <groupId>org.apache.camel.springboot</groupId> + <artifactId>camel-langchain4j-chat-starter</artifactId> +</dependency> +<dependency> + <groupId>org.apache.camel</groupId> + <artifactId>camel-ai-observability</artifactId> +</dependency> +<dependency> + <groupId>dev.langchain4j</groupId> + <artifactId>langchain4j-ollama-spring-boot-starter</artifactId> +</dependency> +<dependency> + <groupId>org.springframework.boot</groupId> + <artifactId>spring-boot-starter-actuator</artifactId> +</dependency> +<dependency> + <groupId>io.micrometer</groupId> + <artifactId>micrometer-registry-prometheus</artifactId> +</dependency> +``` + +### application.properties + +```properties +# LangChain4j / Ollama (auto-configures chatLanguageModel bean) +langchain4j.ollama.chat-model.base-url=http://localhost:11434 +langchain4j.ollama.chat-model.model-name=llama3.2 +langchain4j.ollama.chat-model.temperature=0.2 +langchain4j.ollama.chat-model.timeout=PT120S + +# Camel YAML routes +camel.main.routes-include-pattern=camel/* + +# GenAI observability +camel.ai.observability.enabled=true +camel.opentelemetry2.enabled=true + +# Spring Actuator (Prometheus scrape target) +management.endpoints.web.exposure.include=health,prometheus,info +management.prometheus.metrics.export.enabled=true + +# OTLP export to VictoriaTraces +camel.opentelemetry2.export-target=jaeger +otel.exporter.otlp.endpoint=http://localhost:9428/insert/opentelemetry/v1/traces +otel.exporter.otlp.protocol=http/protobuf +``` + +### The Camel route (YAML) + +```yaml +- route: + id: genai-chat + from: + uri: timer:genai + parameters: + period: "15000" + steps: + - setBody: + constant: "In one sentence, what is Apache Camel integration?" + - to: + uri: langchain4j-chat:demo + parameters: + chatModel: "#chatLanguageModel" + - log: + message: "LLM reply: ${body}" + - log: + message: "Models: req=${header.CamelLangChain4jChatRequestModel} resp=${header.CamelLangChain4jChatResponseModel}" +``` + +## Optional: Camel TUI against Spring Boot + +Uncomment `camel-cli-connector-starter` in `pom.xml` and set `camel.cli.enabled=true`, then run `camel tui`. +You get the same **Spans** tab and **Ctrl+U** AI Usage view as Part 1, but against a production-style runtime. + +## JBang vs Spring Boot + +| Concern | Part 1 (JBang) | Part 2 (Spring Boot) | +|---------|----------------|----------------------| +| Time to first span | Minutes (`camel run --observe`) | Minutes + docker compose | +| Metrics endpoint | `/observe/metrics` on port 9876 | `/actuator/prometheus` on port 8080 | +| Trace UI (dev) | TUI Spans tab (built-in) | VictoriaTraces VMUI | +| Production fit | Prototyping, CI demos | Standard Spring ops (Actuator, K8s probes) | + +Most teams: **prototype in JBang**, **deploy observability pattern in Spring Boot**. Review Comment: prototype in CLI/TUI ########## content/blog/2026/08/camel-genai-observability-jbang/index.md: ########## @@ -0,0 +1,222 @@ +--- +title: "Observe Your Camel AI Routes with GenAI OpenTelemetry" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "opentelemetry", "ollama", "langchain4j", "camel jbang", "camel tui", "camel 4.23", "llm"] +preview: "Apache Camel 4.23 adds GenAI observability — OpenTelemetry spans and Micrometer metrics for LLM routes. Prototype it in minutes with JBang, Ollama, and the Camel TUI." +--- + +Once LLMs live inside Camel routes, the next question is always the same: *how much are we spending, and +where is latency coming from?* Apache Camel **4.23** introduces **GenAI observability** — OpenTelemetry spans +and Micrometer metrics for LLM producers, aligned with the +[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). + +This is **Blog 1** in a two-part series. We start with the fastest path to visible AI telemetry: Camel JBang, +a LangChain4j chat route, [Ollama](https://ollama.com/), and the Camel CLI/TUI — no Spring Boot required. + +Special thanks to **Claus Ibsen** for guidance on shaping this material. + +## The three phases of Camel Gen AI + +| Phase | Name | What you do | +|-------|------|-------------| +| 1 | **Connect** | Wire LLMs into routes (`langchain4j-chat`, `openai`, Spring AI) | +| 2 | **Act** | Give AI tools via `ai-tool:`, agents, MCP server, A2A | +| 3 | **Operate** | Run in production with guardrails, RAG, and **GenAI observability** | + +This post covers Phase 1 plus the first observability prototype. +[Part 2](/blog/2026/08/camel-genai-observability-spring-boot/) moves to Spring Boot and a full observability +stack (Prometheus, VictoriaTraces, Perses). + +## What you'll build + +A timer-driven route that calls Ollama every 15 seconds. For each LLM invocation you get: + +- Micrometer timer: `gen_ai.client.operation` +- Micrometer counter: `gen_ai.client.token.usage` (tags `input` / `output`) +- OpenTelemetry child spans with `gen_ai.request.model`, token counts, finish reason +- Exchange headers: `CamelLangChain4jChatRequestModel`, `CamelLangChain4jChatResponseModel` +- TUI **Spans** tab visualization + AI Usage view (**Ctrl+U**) + +``` +timer:genai (every 15s) + └─ langchain4j-chat (Ollama ChatModel) + ├─ camel-ai-observability-api → start GenAI span + record metrics + ├─ camel-opentelemetry2 → export span attributes + └─ camel-micrometer → gen_ai.client.* metrics + +camel run --observe + └─ camel-observability-services + ├─ /observe/health + ├─ /observe/metrics (Prometheus format) + └─ TUI Spans collector (embedded OTLP for dev) + +camel tui + ├─ Spans tab (shortcut: o) + └─ AI panel + Ctrl+U (CLI ask + route GenAI usage combined) +``` + +## Prerequisites + +- Camel JBang **4.23+** (`camel version`) +- Ollama installed and running + +```shell +ollama pull llama3.2 +ollama serve +``` + +## Run the example + +The bundled example ships with Camel 4.23 at +`dsl/camel-jbang/camel-jbang-core/src/main/resources/examples/ai/genai-observability/` +([source on GitHub](https://github.com/apache/camel/pull/25891/files)). + +```shell +cd ai/genai-observability + +camel run GenAiObservabilityRoute.java application.properties \ + --observe \ + --dependency=camel-langchain4j-chat \ + --dependency=camel-ai-observability \ + --dependency=langchain4j-ollama +``` + +The `--observe` flag adds `camel-observability-services`, enables health checks, Micrometer metrics, +OpenTelemetry tracing, and powers the TUI **Spans** tab on the management port (default `9876`). + +Wait for log lines like: + +```text +LLM reply: Apache Camel is an open source integration framework... +Request model: llama3.2 +Response model: llama3.2 +``` + +## Inspect Prometheus metrics + +```shell +curl -s http://127.0.0.1:9876/observe/metrics | grep gen_ai +``` + +Example output (abbreviated): + +```text +# HELP gen_ai_client_operation GenAI client operation duration +gen_ai_client_operation_count{gen_ai_operation_name="chat",gen_ai_system="langchain4j",...} 3.0 + +# HELP gen_ai_client_token_usage GenAI token usage +gen_ai_client_token_usage_total{gen_ai_token_type="input",...} 42.0 +gen_ai_client_token_usage_total{gen_ai_token_type="output",...} 18.0 +``` + +## Explore spans in the TUI + +Open a second terminal: + +```shell +camel tui +``` + +1. Select the running integration (`genai-observability` or similar) +2. Press **o** or navigate to **More → Spans** +3. Wait for the next timer tick (~15s) and watch a new span appear + +Key span attributes: + +| Attribute | Meaning | +|-----------|---------| +| `gen_ai.operation.name` | e.g. `chat`, `embeddings` | +| `gen_ai.system` | Provider abstraction (e.g. `langchain4j`) | +| `gen_ai.request.model` | Model requested | +| `gen_ai.response.model` | Model that served the response | +| `gen_ai.usage.input_tokens` | Prompt tokens | +| `gen_ai.usage.output_tokens` | Completion tokens | +| `camel.component` | e.g. `langchain4j-chat` | + +Press **Ctrl+U** in the AI panel to toggle the **AI Usage** view — token consumption from both +TUI `camel ask` prompts and route LLM calls on one screen. Review Comment: It would be great if you took a screenshot from the TUI showing this and including here in the blog ########## content/blog/2026/08/camel-genai-observability-jbang/index.md: ########## @@ -0,0 +1,222 @@ +--- +title: "Observe Your Camel AI Routes with GenAI OpenTelemetry" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "opentelemetry", "ollama", "langchain4j", "camel jbang", "camel tui", "camel 4.23", "llm"] +preview: "Apache Camel 4.23 adds GenAI observability — OpenTelemetry spans and Micrometer metrics for LLM routes. Prototype it in minutes with JBang, Ollama, and the Camel TUI." +--- + +Once LLMs live inside Camel routes, the next question is always the same: *how much are we spending, and +where is latency coming from?* Apache Camel **4.23** introduces **GenAI observability** — OpenTelemetry spans +and Micrometer metrics for LLM producers, aligned with the +[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). + +This is **Blog 1** in a two-part series. We start with the fastest path to visible AI telemetry: Camel JBang, +a LangChain4j chat route, [Ollama](https://ollama.com/), and the Camel CLI/TUI — no Spring Boot required. + +Special thanks to **Claus Ibsen** for guidance on shaping this material. + +## The three phases of Camel Gen AI + +| Phase | Name | What you do | +|-------|------|-------------| +| 1 | **Connect** | Wire LLMs into routes (`langchain4j-chat`, `openai`, Spring AI) | +| 2 | **Act** | Give AI tools via `ai-tool:`, agents, MCP server, A2A | +| 3 | **Operate** | Run in production with guardrails, RAG, and **GenAI observability** | + +This post covers Phase 1 plus the first observability prototype. +[Part 2](/blog/2026/08/camel-genai-observability-spring-boot/) moves to Spring Boot and a full observability +stack (Prometheus, VictoriaTraces, Perses). + +## What you'll build + +A timer-driven route that calls Ollama every 15 seconds. For each LLM invocation you get: + +- Micrometer timer: `gen_ai.client.operation` +- Micrometer counter: `gen_ai.client.token.usage` (tags `input` / `output`) +- OpenTelemetry child spans with `gen_ai.request.model`, token counts, finish reason +- Exchange headers: `CamelLangChain4jChatRequestModel`, `CamelLangChain4jChatResponseModel` +- TUI **Spans** tab visualization + AI Usage view (**Ctrl+U**) + +``` +timer:genai (every 15s) + └─ langchain4j-chat (Ollama ChatModel) + ├─ camel-ai-observability-api → start GenAI span + record metrics + ├─ camel-opentelemetry2 → export span attributes + └─ camel-micrometer → gen_ai.client.* metrics + +camel run --observe + └─ camel-observability-services + ├─ /observe/health + ├─ /observe/metrics (Prometheus format) + └─ TUI Spans collector (embedded OTLP for dev) + +camel tui + ├─ Spans tab (shortcut: o) + └─ AI panel + Ctrl+U (CLI ask + route GenAI usage combined) +``` + +## Prerequisites + +- Camel JBang **4.23+** (`camel version`) +- Ollama installed and running + +```shell +ollama pull llama3.2 +ollama serve +``` + +## Run the example + +The bundled example ships with Camel 4.23 at +`dsl/camel-jbang/camel-jbang-core/src/main/resources/examples/ai/genai-observability/` +([source on GitHub](https://github.com/apache/camel/pull/25891/files)). + +```shell +cd ai/genai-observability + +camel run GenAiObservabilityRoute.java application.properties \ Review Comment: We should make camel jbang able to auto discover these dependencies so you dont need to add these. Create a JIRA about that. ########## content/blog/2026/08/camel-genai-observability-spring-boot/index.md: ########## @@ -0,0 +1,274 @@ +--- +title: "GenAI Observability with Spring Boot and the Camel Observability Stack" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "spring boot", "prometheus", "opentelemetry", "victoriatraces", "perses", "ollama", "langchain4j", "camel 4.23"] +preview: "Take Camel GenAI observability from JBang prototypes to production — Spring Boot Actuator, Prometheus scraping, OTLP traces in VictoriaTraces, and optional Perses dashboards." +--- + +In [Part 1](/blog/2026/08/camel-genai-observability-jbang/) we prototyped GenAI observability with Camel JBang +and the TUI. This follow-up — **Phase 3 (Operate)** — shows the same `gen_ai.*` telemetry in a **Spring Boot** +application wired to the observability stack Camel uses in test-infra: **Prometheus**, **VictoriaTraces**, and **Perses**. + +Special thanks to **Claus Ibsen** for guidance on this series. + +The runnable sample lives in +[`spring-boot/genai-observability`](https://github.com/apache/camel-examples/pull/255/files) on camel-examples +(open PR until merged to `main`). + +## Architecture + +``` +┌─────────────────┐ scrape ┌──────────────┐ +│ Spring Boot │ ──────────────► │ Prometheus │ +│ Camel + LLM │ │ :9090 │ +│ :8080 │ └──────┬───────┘ +└────────┬────────┘ │ + │ OTLP traces ▼ + ▼ ┌──────────────┐ +┌─────────────────┐ │ Perses │ +│ VictoriaTraces │ ◄── dashboards │ :8088 │ +│ :9428 │ └──────────────┘ +└─────────────────┘ + +Optional: camel tui ──► Spring Boot via cli-connector +``` + +For every LLM call in the YAML route: + +- Actuator `/actuator/prometheus` exposes `gen_ai_client_*` metrics +- OTLP export sends spans with token attributes to VictoriaTraces +- You query traces by `gen_ai.operation.name="chat"` + +## Quick start + +### Prerequisites + +```shell +java -version # 17+ +mvn -version # 3.9+ +docker --version # for observability stack +ollama pull llama3.2 +ollama serve +``` + +### Start the observability stack + +From the example directory: + +```shell Review Comment: Take a look at Camel JBang where we have `camel infra` it has an obs stack you can run easily as well. It may be nice to use that, or make this docker compose use same port / credeitnials as the camel infra does. And you can also start the infra in TUI ########## content/blog/2026/08/camel-genai-observability-spring-boot/index.md: ########## @@ -0,0 +1,274 @@ +--- +title: "GenAI Observability with Spring Boot and the Camel Observability Stack" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "spring boot", "prometheus", "opentelemetry", "victoriatraces", "perses", "ollama", "langchain4j", "camel 4.23"] +preview: "Take Camel GenAI observability from JBang prototypes to production — Spring Boot Actuator, Prometheus scraping, OTLP traces in VictoriaTraces, and optional Perses dashboards." +--- + +In [Part 1](/blog/2026/08/camel-genai-observability-jbang/) we prototyped GenAI observability with Camel JBang +and the TUI. This follow-up — **Phase 3 (Operate)** — shows the same `gen_ai.*` telemetry in a **Spring Boot** +application wired to the observability stack Camel uses in test-infra: **Prometheus**, **VictoriaTraces**, and **Perses**. + +Special thanks to **Claus Ibsen** for guidance on this series. Review Comment: Same as before ########## content/blog/2026/08/camel-genai-observability-spring-boot/index.md: ########## @@ -0,0 +1,274 @@ +--- +title: "GenAI Observability with Spring Boot and the Camel Observability Stack" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "spring boot", "prometheus", "opentelemetry", "victoriatraces", "perses", "ollama", "langchain4j", "camel 4.23"] +preview: "Take Camel GenAI observability from JBang prototypes to production — Spring Boot Actuator, Prometheus scraping, OTLP traces in VictoriaTraces, and optional Perses dashboards." +--- + +In [Part 1](/blog/2026/08/camel-genai-observability-jbang/) we prototyped GenAI observability with Camel JBang +and the TUI. This follow-up — **Phase 3 (Operate)** — shows the same `gen_ai.*` telemetry in a **Spring Boot** +application wired to the observability stack Camel uses in test-infra: **Prometheus**, **VictoriaTraces**, and **Perses**. + +Special thanks to **Claus Ibsen** for guidance on this series. + +The runnable sample lives in +[`spring-boot/genai-observability`](https://github.com/apache/camel-examples/pull/255/files) on camel-examples +(open PR until merged to `main`). + +## Architecture + +``` +┌─────────────────┐ scrape ┌──────────────┐ +│ Spring Boot │ ──────────────► │ Prometheus │ +│ Camel + LLM │ │ :9090 │ +│ :8080 │ └──────┬───────┘ +└────────┬────────┘ │ + │ OTLP traces ▼ + ▼ ┌──────────────┐ +┌─────────────────┐ │ Perses │ +│ VictoriaTraces │ ◄── dashboards │ :8088 │ +│ :9428 │ └──────────────┘ +└─────────────────┘ + +Optional: camel tui ──► Spring Boot via cli-connector +``` + +For every LLM call in the YAML route: Review Comment: Oh this is a bit opposite the non spring example is java, and this is Java. SB tend to be for java devs. ########## content/blog/2026/08/camel-genai-observability-spring-boot/index.md: ########## @@ -0,0 +1,274 @@ +--- +title: "GenAI Observability with Spring Boot and the Camel Observability Stack" +date: 2026-08-29 +draft: false +authors: [atiaomar1978-hub] +categories: ["AI", "Howtos"] +keywords: ["apache camel", "genai", "observability", "spring boot", "prometheus", "opentelemetry", "victoriatraces", "perses", "ollama", "langchain4j", "camel 4.23"] +preview: "Take Camel GenAI observability from JBang prototypes to production — Spring Boot Actuator, Prometheus scraping, OTLP traces in VictoriaTraces, and optional Perses dashboards." +--- + +In [Part 1](/blog/2026/08/camel-genai-observability-jbang/) we prototyped GenAI observability with Camel JBang +and the TUI. This follow-up — **Phase 3 (Operate)** — shows the same `gen_ai.*` telemetry in a **Spring Boot** +application wired to the observability stack Camel uses in test-infra: **Prometheus**, **VictoriaTraces**, and **Perses**. + +Special thanks to **Claus Ibsen** for guidance on this series. + +The runnable sample lives in +[`spring-boot/genai-observability`](https://github.com/apache/camel-examples/pull/255/files) on camel-examples +(open PR until merged to `main`). + +## Architecture + +``` +┌─────────────────┐ scrape ┌──────────────┐ +│ Spring Boot │ ──────────────► │ Prometheus │ +│ Camel + LLM │ │ :9090 │ +│ :8080 │ └──────┬───────┘ +└────────┬────────┘ │ + │ OTLP traces ▼ + ▼ ┌──────────────┐ +┌─────────────────┐ │ Perses │ +│ VictoriaTraces │ ◄── dashboards │ :8088 │ +│ :9428 │ └──────────────┘ +└─────────────────┘ + +Optional: camel tui ──► Spring Boot via cli-connector +``` + +For every LLM call in the YAML route: + +- Actuator `/actuator/prometheus` exposes `gen_ai_client_*` metrics +- OTLP export sends spans with token attributes to VictoriaTraces +- You query traces by `gen_ai.operation.name="chat"` + +## Quick start + +### Prerequisites + +```shell +java -version # 17+ +mvn -version # 3.9+ +docker --version # for observability stack +ollama pull llama3.2 +ollama serve +``` + +### Start the observability stack + +From the example directory: + +```shell +cd spring-boot/genai-observability +docker compose up -d +``` + +| Service | Port | Role | +|---------|------|------| +| Prometheus | 9090 | Scrapes Spring Boot Actuator Prometheus endpoint | +| VictoriaTraces | 9428 | Stores OTLP traces; UI at `/select/vmui` | +| Perses | 8088 | Metrics dashboards (optional visualization) | + +Container images match `camel-test-infra-observability` so CI and blog readers use the same stack. + +### Run Spring Boot + +```shell +mvn spring-boot:run +``` + +Wait for: + +```text +Started GenAiObservabilityApplication +LLM reply: Apache Camel is an integration framework... +Models: req=llama3.2 resp=llama3.2 +``` + +## Verify GenAI metrics + +```shell +curl -s http://localhost:8080/actuator/prometheus | grep gen_ai +``` + +Expected Micrometer names: + +- `gen_ai_client_operation` — timer/summary of LLM call duration +- `gen_ai_client_token_usage` — counter with tag `gen_ai_token_type=input|output` + +### Prometheus queries + +Open [http://localhost:9090](http://localhost:9090) and try: + +```promql +# Total output tokens over time +increase(gen_ai_client_token_usage_total{gen_ai_token_type="output"}[1h]) + +# LLM call rate +rate(gen_ai_client_operation_count[5m]) + +# By Camel component +sum by (camel_component) (gen_ai_client_operation_count) +``` + +### Example alert rule (sketch) + +```yaml +groups: + - name: camel-genai + rules: + - alert: HighGenAITokenBurn + expr: rate(gen_ai_client_token_usage_total[15m]) > 1000 + for: 10m + labels: + severity: warning + annotations: + summary: "High GenAI token usage on {{ $labels.job }}" +``` + +## Explore traces in VictoriaTraces + +Open [http://localhost:9428/select/vmui](http://localhost:9428/select/vmui) + +Useful trace search filters: + +- `gen_ai.operation.name="chat"` +- `gen_ai.request.model="llama3.2"` +- `camel.component="langchain4j-chat"` + +Each span carries `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, +`gen_ai.response.finish_reasons`, and `gen_ai.system` (e.g. `langchain4j`). + +Correlate traces with logs by passing the trace ID from MDC — see the +[OpenTelemetry docs](/manual/camel-jbang-tui.html#_opentelemetry_spans). + +## Key configuration + +### Maven dependencies (excerpt) + +```xml +<dependency> + <groupId>org.apache.camel.springboot</groupId> + <artifactId>camel-spring-boot-starter</artifactId> +</dependency> +<dependency> + <groupId>org.apache.camel.springboot</groupId> + <artifactId>camel-observability-services-starter</artifactId> +</dependency> +<dependency> + <groupId>org.apache.camel.springboot</groupId> + <artifactId>camel-langchain4j-chat-starter</artifactId> +</dependency> +<dependency> + <groupId>org.apache.camel</groupId> + <artifactId>camel-ai-observability</artifactId> +</dependency> +<dependency> + <groupId>dev.langchain4j</groupId> + <artifactId>langchain4j-ollama-spring-boot-starter</artifactId> +</dependency> +<dependency> + <groupId>org.springframework.boot</groupId> + <artifactId>spring-boot-starter-actuator</artifactId> +</dependency> +<dependency> + <groupId>io.micrometer</groupId> + <artifactId>micrometer-registry-prometheus</artifactId> +</dependency> +``` + +### application.properties + +```properties +# LangChain4j / Ollama (auto-configures chatLanguageModel bean) +langchain4j.ollama.chat-model.base-url=http://localhost:11434 +langchain4j.ollama.chat-model.model-name=llama3.2 +langchain4j.ollama.chat-model.temperature=0.2 +langchain4j.ollama.chat-model.timeout=PT120S + +# Camel YAML routes +camel.main.routes-include-pattern=camel/* + +# GenAI observability +camel.ai.observability.enabled=true +camel.opentelemetry2.enabled=true + +# Spring Actuator (Prometheus scrape target) +management.endpoints.web.exposure.include=health,prometheus,info +management.prometheus.metrics.export.enabled=true + +# OTLP export to VictoriaTraces +camel.opentelemetry2.export-target=jaeger +otel.exporter.otlp.endpoint=http://localhost:9428/insert/opentelemetry/v1/traces +otel.exporter.otlp.protocol=http/protobuf +``` + +### The Camel route (YAML) + +```yaml +- route: + id: genai-chat + from: + uri: timer:genai + parameters: + period: "15000" + steps: + - setBody: + constant: "In one sentence, what is Apache Camel integration?" + - to: + uri: langchain4j-chat:demo + parameters: + chatModel: "#chatLanguageModel" + - log: + message: "LLM reply: ${body}" + - log: + message: "Models: req=${header.CamelLangChain4jChatRequestModel} resp=${header.CamelLangChain4jChatResponseModel}" +``` + +## Optional: Camel TUI against Spring Boot + +Uncomment `camel-cli-connector-starter` in `pom.xml` and set `camel.cli.enabled=true`, then run `camel tui`. +You get the same **Spans** tab and **Ctrl+U** AI Usage view as Part 1, but against a production-style runtime. + +## JBang vs Spring Boot + +| Concern | Part 1 (JBang) | Part 2 (Spring Boot) | +|---------|----------------|----------------------| +| Time to first span | Minutes (`camel run --observe`) | Minutes + docker compose | +| Metrics endpoint | `/observe/metrics` on port 9876 | `/actuator/prometheus` on port 8080 | +| Trace UI (dev) | TUI Spans tab (built-in) | VictoriaTraces VMUI | +| Production fit | Prototyping, CI demos | Standard Spring ops (Actuator, K8s probes) | + +Most teams: **prototype in JBang**, **deploy observability pattern in Spring Boot**. + +## Production checklist + +1. Set `camel.ai.observability.enabled=true` explicitly in all environments Review Comment: is it `camel.aiObservability...` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
