This is an automated email from the ASF dual-hosted git repository.
jamesnetherton pushed a commit to branch camel-quarkus-main
in repository https://gitbox.apache.org/repos/asf/camel-quarkus-examples.git
The following commit(s) were added to refs/heads/camel-quarkus-main by this
push:
new 2c268319 Fix Observability example dev mode data export to LGTM
services
2c268319 is described below
commit 2c268319231b2103f4962b0cc7b959eacc4618ba
Author: James Netherton <[email protected]>
AuthorDate: Tue Sep 22 09:58:10 2026 +0100
Fix Observability example dev mode data export to LGTM services
---
observability/README.adoc | 154 +++++++++------------
.../devmode/DevModeMetricsEndpoint.java | 53 +++++++
...ana-dashboard-camel-quarkus-observability.json} | 0
.../src/main/resources/application.properties | 30 +++-
4 files changed, 143 insertions(+), 94 deletions(-)
diff --git a/observability/README.adoc b/observability/README.adoc
index 6200428b..59b1266c 100644
--- a/observability/README.adoc
+++ b/observability/README.adoc
@@ -19,6 +19,10 @@ workspace. Any modifications in your project will
automatically take effect in t
TIP: Please refer to the Development mode section of
https://camel.apache.org/camel-quarkus/latest/first-steps.html#_development_mode[Camel
Quarkus User guide] for more details.
+NOTE: Dev mode requires a working container runtime such as Docker or Podman.
The Quarkus Observability dev service
+uses it to run a Grafana OTel LGTM container, which the application exports
traces to. See the <<_tracing,Tracing>>
+section for details. Set `quarkus.observability.enabled=false` if you would
rather not have the container started.
+
=== Camel Observability Services
This project includes the `camel-quarkus-observability-services` extension.
@@ -121,6 +125,9 @@ and you should see 3 lines of different metrics (with the
same value, as they ar
NOTE: Maybe you've noticed the Prometheus output format. If you would rather
use the JSON format, please follow the Quarkus Micrometer management interface
https://quarkus.io/guides/micrometer#management-interface[configuration guide].
+In dev mode only, the same registry is also served from
`localhost:8080/q/metrics`, so that the Grafana dev service
+can scrape it. See <<_dev_mode_metrics,Dev mode metrics>>.
+
=== Health endpoint
Camel provides some out of the box liveness and readiness checks. To see this
working, interrogate the `/observe/health/live` and `/observe/health/ready`
endpoints on port `9876`:
@@ -146,99 +153,73 @@ You can also directly leverage MicroProfile Health APIs
to create checks. Class
To be able to diagnose problems in Camel Quarkus applications, it's useful to
instrument method calls, HTTP interactions etc with OpenTelemetry.
-This example is pre-configured to send traces to Tempo via OTLP on
`http://localhost:4317`. The behavior differs based on how you run the
application:
-
-==== Dev Mode (quarkus:dev)
-
-When running `mvn clean compile quarkus:dev`, the Quarkus Observability dev
service is automatically enabled (see
https://quarkus.io/guides/observability-devservices-lgtm[Observability Dev
Services with Grafana OTel LGTM]).
-
-The dev service automatically:
-- Starts a Grafana LGTM container with Tempo for tracing
-- Overrides the `quarkus.otel.exporter.otlp.traces.endpoint` property to point
to the dev service container
-- Assigns a random port to Grafana (check logs for
`grafana.endpoint=http://localhost:XXXXX`)
-
-To view traces in dev mode, find the Grafana endpoint in logs and browse to it.
-
-==== Docker Compose Mode (Recommended)
-
-When running the packaged application with Docker Compose (see Grafana
Dashboards section above), traces are sent to the Tempo container at
`localhost:4317`.
-
-The OpenTelemetry configuration in `application.properties` is already set:
+The OpenTelemetry SDK is disabled by default by
`camel-quarkus-observability-services`, so `application.properties` enables it:
-[source, text]
+[source, properties]
----
-quarkus.otel.exporter.otlp.traces.endpoint = http://localhost:4317
quarkus.otel.sdk.disabled=false
----
-Traces are automatically visible in Grafana at http://localhost:3000.
+Note that no OTLP exporter endpoint is configured. Where the traces are sent
depends on how the application is run.
-==== Cloud/Kubernetes Deployment
+==== Dev mode
-For cloud or Kubernetes deployments, override the OTLP endpoint using an
environment variable:
+In dev mode, the
https://quarkus.io/guides/observability-devservices-lgtm[Grafana OTel LGTM dev
service] starts a
+container running Grafana, Tempo, Loki, Prometheus and an OpenTelemetry
collector, and configures the application to
+export to it.
-[source, shell]
-----
-export QUARKUS_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://tempo-service:4317
-----
+NOTE: Do not set `quarkus.otel.exporter.otlp.endpoint` (or its per signal
variants) in `application.properties`. It
+takes precedence over what the dev service configures, and traces then go
nowhere.
+
+Grafana is at http://localhost:3000 (`admin` / `admin`). The port is pinned
with
+`%dev.quarkus.observability.lgtm.grafana-port` so that it does not change on
every restart.
-Or use property placeholders in `application.properties`:
+To view traces, choose *Explore*, select the *Tempo* data source and search
for `{name="POST /greeting"}` - the timer
+route produces one of those every 10 seconds. Requests you send yourself show
up as `{name="GET /greeting"}`:
-[source, text]
+[source,shell]
----
-quarkus.otel.exporter.otlp.traces.endpoint =
http://${TEMPO_HOST:localhost}:4317
+curl -s localhost:8080/greeting
----
-NOTE: For information about other OpenTelemetry exporters, refer to the Camel
Quarkus OpenTelemetry
https://camel.apache.org/camel-quarkus/next/reference/extensions/opentelemetry.html#extensions-opentelemetry-usage-exporters[extension
documentation].
-
-==== Understanding Traces
-
-The `platform-http` consumer route introduces a random delay to simulate
latency, hence the overall time of each trace should be different. When viewing
a trace, you should see
-a hierarchy of 8 spans showing the progression of the message exchange through
each endpoint.
-
-=== Grafana Dashboards
-
-This example includes a pre-configured Grafana dashboard to visualize metrics,
distributed traces, and JVM statistics.
+Metrics and logs land in the same Grafana instance, see the
<<_grafana_dashboard,Grafana dashboard>> section.
-==== Quick Start with Docker Compose (Recommended)
+==== Package and run
-The easiest way to visualize the dashboard with working metrics is using
Docker Compose with static ports:
+When the packaged application is run outside of dev mode, there is no dev
service and the Quarkus defaults apply, which
+is OTLP over gRPC to `http://localhost:4317`. Point the application at your
own collector, Tempo or Jaeger instance with:
[source,shell]
----
-# Start the observability stack (Grafana, Prometheus, Tempo)
-docker compose -f src/main/docker/docker-compose-grafana.yml up -d
-
-# In another terminal, package and run the application
-mvn clean package -DskipTests
-java -jar target/quarkus-app/quarkus-run.jar
+export QUARKUS_OTEL_EXPORTER_OTLP_ENDPOINT=http://my-collector:4317
----
-Access Grafana at http://localhost:3000 (credentials: `admin/admin`). The
dashboard is automatically provisioned and ready to use!
-
-NOTE: To stop the observability stack: `docker compose -f
src/main/docker/docker-compose-grafana.yml down -v`
-
-==== Alternative: Dev Mode with Manual Dashboard Import
-
-When running in dev mode, Quarkus automatically provisions a Grafana instance
with dynamic ports:
+Or configure it in `application.properties`:
-[source,shell]
+[source, properties]
----
-mvn clean compile quarkus:dev
+quarkus.otel.exporter.otlp.endpoint =
http://${OTEL_COLLECTOR_HOST:localhost}:4317
----
-IMPORTANT: The LGTM dev services Grafana can display *distributed traces* but
cannot display *metrics* from the dashboard panels. This is because the
Prometheus instance inside the LGTM container cannot scrape metrics from
`localhost:9876` on the host machine. For full dashboard functionality, use the
Docker Compose setup above.
+NOTE: For information about other OpenTelemetry exporters, refer to the Camel
Quarkus OpenTelemetry
https://camel.apache.org/camel-quarkus/next/reference/extensions/opentelemetry.html#extensions-opentelemetry-usage-exporters[extension
documentation].
+
+==== Understanding traces
-To view traces in dev mode:
+The `platform-http` consumer route introduces a random delay to simulate
latency, hence the overall time of each trace
+should be different. When viewing a trace, you should see a hierarchy of spans
showing the progression of the message
+exchange, from the inbound `platform-http` request, through the HTTP call to
the `/greeting-provider` endpoint, down to
+the individual Camel processors.
-1. Find the Grafana endpoint in logs: `grafana.endpoint=http://localhost:XXXXX`
-2. Access Grafana (credentials: `admin/admin`)
-3. Navigate to *Explore* → Select *Tempo* datasource
-4. Query traces with: `{name="POST /greeting"}`
+=== Grafana dashboard
-==== Dashboard Contents
+This example ships a Grafana dashboard at
+link:src/main/resources/META-INF/grafana/grafana-dashboard-camel-quarkus-observability.json[grafana-dashboard-camel-quarkus-observability.json].
+Any dashboard placed in `META-INF/grafana` is provisioned automatically into
the Grafana instance of the LGTM dev
+service, so in dev mode it is available straight away, listed as *Camel
Quarkus Observability Dashboard*.
-The Camel Quarkus Observability Dashboard includes four main sections:
+image::docs/images/grafana-dashboard.png[Grafana Dashboard showing Camel
Quarkus observability metrics]
+
+It is organised in four sections:
*Custom Application Metrics*
@@ -254,7 +235,7 @@ The Camel Quarkus Observability Dashboard includes four
main sections:
*Distributed Tracing*
-- *Distributed Traces*: Information panel with link to Tempo Explore for
viewing detailed traces
+- *Distributed Traces*: Information panel describing how to view traces in
Tempo
*JVM & System Metrics*
@@ -263,39 +244,30 @@ The Camel Quarkus Observability Dashboard includes four
main sections:
- *Garbage Collection Pause Time*: GC pause duration by type
- *Active HTTP Connections*: Current number of active requests
-==== What You'll See
+The dashboard auto-refreshes every 5 seconds. The timer route drives some of
the metrics on its own, and you can
+generate more traffic with `curl localhost:8080/greeting` in a loop.
-image::docs/images/grafana-dashboard.png[Grafana Dashboard showing Camel
Quarkus observability metrics]
+==== Dev mode metrics
-When the application is running, the dashboard will show:
+The dev service can only scrape metrics from the application HTTP port,
whereas `camel-quarkus-observability-services`
+serves them from the management interface. To bridge that gap,
+link:src/main/java/org/acme/observability/devmode/DevModeMetricsEndpoint.java[DevModeMetricsEndpoint.java]
exposes the
+same registry on `/q/metrics` in dev mode. It is not part of the packaged
application.
-- *Timer Counter*: Increments automatically every 10 seconds (from the timer
route)
-- *Greeting Metrics*: Increase when you call the endpoint:
-+
-[source,shell]
+IMPORTANT: On Linux, uncomment the following in `application.properties`,
otherwise the dev service cannot reach the
+application and the metrics panels stay empty.
+
+[source, properties]
----
-curl localhost:8080/greeting
+%dev.quarkus.http.host = 0.0.0.0
----
-- *Distributed Traces*: Click "Open Tempo Explore" to view complete trace
hierarchy with 8 spans showing message flow through Camel endpoints
-- *JVM Metrics*: Real-time memory, CPU, and GC statistics
-
-TIP: The dashboard auto-refreshes every 5 seconds. Generate some traffic using
`curl` in a loop to see the metrics change in real-time.
-
-==== Docker Compose Details
-
-The Docker Compose setup includes:
-
-- *Grafana* on http://localhost:3000 with the dashboard automatically
provisioned
-- *Prometheus* on http://localhost:9090 scraping metrics from the application
every 5 seconds
-- *Tempo* on http://localhost:3200 receiving distributed traces via OTLP
+==== Logs
-The application exposes:
-- Metrics at `http://localhost:9876/observe/metrics` (scraped by Prometheus)
-- Traces sent to Tempo at `http://localhost:4317` (OTLP gRPC)
-- Health checks at `http://localhost:9876/observe/health/live` and
`/observe/health/ready`
+`quarkus.otel.logs.enabled=true` exports log records over OTLP as well. They
are searchable in Loki via *Explore* or
+the bundled *Quarkus OpenTelemetry Logging* dashboard, and each record carries
the `trace_id` and `span_id` of the
+exchange that produced it.
-All services use **static ports** for easy access and documentation.
=== Jolokia & Hawtio
diff --git
a/observability/src/main/java/org/acme/observability/devmode/DevModeMetricsEndpoint.java
b/observability/src/main/java/org/acme/observability/devmode/DevModeMetricsEndpoint.java
new file mode 100644
index 00000000..4ce7899c
--- /dev/null
+++
b/observability/src/main/java/org/acme/observability/devmode/DevModeMetricsEndpoint.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.acme.observability.devmode;
+
+import io.micrometer.prometheus.PrometheusMeterRegistry;
+import io.quarkus.arc.profile.IfBuildProfile;
+import io.vertx.ext.web.Router;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.enterprise.event.Observes;
+import jakarta.inject.Inject;
+
+/**
+ * Exposes the Prometheus registry on the main HTTP port, in dev mode only.
+ *
+ * <p>
+ * The Grafana LGTM dev service configures its Prometheus to scrape {@code
quarkus.http.port} at
+ * {@code quarkus.management.root-path} + {@code /metrics}. It has no support
for the Quarkus management interface,
+ * which is what {@code camel-quarkus-observability-services} enables and
serves metrics from (port 9876, path
+ * {@code /observe/metrics}), so the scrape target would otherwise never come
up and every metrics panel in Grafana
+ * would stay empty.
+ *
+ * <p>
+ * Serving the same registry from {@code /q/metrics} on the application port
gives the dev service something to
+ * scrape, without moving the endpoint that the rest of this example
demonstrates. The bean is not built into the
+ * application outside of dev mode.
+ */
+@IfBuildProfile("dev")
+@ApplicationScoped
+public class DevModeMetricsEndpoint {
+
+ @Inject
+ PrometheusMeterRegistry registry;
+
+ void registerMetricsRoute(@Observes Router router) {
+ router.get("/q/metrics").handler(ctx -> ctx.response()
+ .putHeader("Content-Type", "text/plain; version=0.0.4;
charset=utf-8")
+ .end(registry.scrape()));
+ }
+}
diff --git
a/observability/src/main/resources/grafana/dashboards/camel-observability-dashboard.json
b/observability/src/main/resources/META-INF/grafana/grafana-dashboard-camel-quarkus-observability.json
similarity index 100%
rename from
observability/src/main/resources/grafana/dashboards/camel-observability-dashboard.json
rename to
observability/src/main/resources/META-INF/grafana/grafana-dashboard-camel-quarkus-observability.json
diff --git a/observability/src/main/resources/application.properties
b/observability/src/main/resources/application.properties
index 1293735a..99d93ff0 100644
--- a/observability/src/main/resources/application.properties
+++ b/observability/src/main/resources/application.properties
@@ -28,11 +28,35 @@ quarkus.application.name = camel-quarkus-observability
# Enables optional tracing of each Camel processor
# quarkus.camel.opentelemetry2.trace-processors=true
-# For OTLP
-quarkus.otel.exporter.otlp.traces.endpoint = http://localhost:4317
-quarkus.otel.exporter.otlp.traces.timeout = 30s
+# For OTLP.
+# Note that the OTLP exporter endpoint and protocol are deliberately not
configured here. In dev mode, the Grafana
+# LGTM dev service sets quarkus.otel.exporter.otlp.endpoint &
quarkus.otel.exporter.otlp.protocol automatically, so
+# that traces are exported to the container it manages. When running the
packaged application, the Quarkus defaults
+# apply (gRPC on http://localhost:4317) and can be overridden for your own
collector, for example with:
+#
+# export QUARKUS_OTEL_EXPORTER_OTLP_ENDPOINT=http://my-collector:4317
+#
+quarkus.otel.exporter.otlp.timeout = 30s
# To enable tracing (it is disabled by default via
camel-quarkus-observability-services)
quarkus.otel.sdk.disabled=false
+# Export log records so that they can be correlated with traces
+quarkus.otel.logs.enabled=true
+
+#
+# Grafana LGTM dev service
+#
+# Bind the Grafana UI to a fixed port, instead of the random one it would get
by default
+%dev.quarkus.observability.lgtm.grafana-port = 3000
+
+# The Prometheus instance of the dev service scrapes the application from
within its container, using a Host header of
+# host.docker.internal. Dev mode binds the HTTP server to localhost, which
makes Quarkus reject any request whose Host
+# header is not a localhost name with a 400 response, so the scrape target has
to be allowed explicitly.
+%dev.quarkus.http.host-validation.allowed-hosts =
localhost,host.docker.internal
+
+# On macOS and Windows the above is enough, as host.docker.internal is proxied
through to the host loopback. On Linux
+# it resolves to the container gateway address instead, which cannot reach a
server bound to localhost, so there the
+# HTTP server has to listen on a non loopback address as well. Uncomment the
following line when developing on Linux.
+# %dev.quarkus.http.host = 0.0.0.0
#
# Camel
