[
https://issues.apache.org/jira/browse/CAMEL-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Federico Mariani updated CAMEL-24051:
-------------------------------------
Description:
h3. Problem / Motivation
Camel's local dev loop ({{camel run --dev}}) and the Kubernetes export path
already wire up {{camel-observability-services}} (Micrometer/Prometheus
metrics, health checks, OpenTelemetry tracing via the bundled
{{camel-opentelemetry2}}) when targeting Kubernetes/OpenShift. However, there
is no local backend to receive or visualize that telemetry: {{camel infra run}}
can start backing services like Postgres or Artemis, but nothing analogous
exists for observability.
As a result, developers only find out whether traces, metrics, and logs
actually work correctly (correct span correlation, meaningful dashboards, sane
metric cardinality, etc.) once the application is deployed to a real OpenShift
cluster, which typically provides the Red Hat build of OpenTelemetry, the Tempo
Operator, Loki/OpenShift Logging, and the Cluster Observability Operator
(Grafana). Telemetry problems surface late, in a shared cluster environment,
rather than at dev/E2E time on a laptop or in CI.
h3. Licensing note (why not grafana/otel-lgtm)
The obvious candidate for this is Grafana's {{grafana/otel-lgtm}} all-in-one
image. It was *ruled out*: Grafana, Loki, and Tempo were relicensed from Apache
2.0 to AGPLv3 in 2021. AGPLv3 is Category X on the ASF's third-party license
policy, so an ASF project cannot depend on it even as opt-in test/dev tooling
without a real [email protected] sign-off, and Red Hat's OSPO treats
AGPL as off-limits even more strictly, including for test-only tooling in
downstream builds. Only Prometheus and the OpenTelemetry Collector in that
bundle are actually Apache 2.0.
h3. Proposed Solution -- Apache-2.0-only stack (validated with a working PoC,
see below)
Use an equivalent stack built entirely from Apache-2.0-licensed components:
* *Metrics*: Prometheus (unchanged -- already Apache 2.0, already the
{{camel-observability-services}} default via {{/observe/metrics}}).
* *Traces*: https://github.com/VictoriaMetrics/VictoriaTraces[VictoriaTraces]
(Apache 2.0) -- ingests OTLP natively and exposes a Jaeger Query
Service-compatible JSON API. Functionally a drop-in swap for Tempo.
* *Logs*: https://github.com/VictoriaMetrics/VictoriaLogs[VictoriaLogs] (Apache
2.0) -- LogsQL query language, accepts OTLP logs directly via
{{/insert/opentelemetry/v1/logs}}. Swap for Loki.
* *Dashboard*: https://perses.dev[Perses] (Apache 2.0, CNCF sandbox project) --
dashboard-as-code tool backed by Amadeus/SAP/Red Hat, and notably *already
powers OpenShift's own new traces UI*, which makes it a much safer downstream
bet than Grafana. Its default plugin set already ships Prometheus, Jaeger, and
VictoriaLogs datasource kinds with no extra plugin install.
Adding this also revisits the fate of the existing {{camel-test-infra-jaeger}}
module (added in CAMEL-23641): since VictoriaTraces exposes a Jaeger-compatible
query API, {{camel-test-infra-jaeger}} does *not* need to be deprecated as
originally floated -- it can keep serving its current role, or the new service
can point at it directly, instead of being replaced outright. This should be
decided during implementation, not assumed here.
h3. Signal-level design rationale (matching real K8s/OCP behavior where it
matters)
* *Traces -- push*: matches OCP's own model (Red Hat build of OpenTelemetry
pushes OTLP spans to a Collector). No divergence between local and cluster
behavior.
* *Metrics -- pull*: OCP's real default is Prometheus scraping via
{{ServiceMonitor}}/{{PodMonitor}}, not app-side push.
{{camel-observability-services}}' existing Prometheus-pull default already
matches this -- the local service's job is to supply a scrape target pointed at
the running app, mirroring what a {{ServiceMonitor}} does on-cluster, *not* to
switch metrics to OTLP push.
* *Logs -- local shortcut, not literal OCP parity*: real OpenShift logging does
not have the app push logs at all -- Vector tails container stdout/stderr and
ships to LokiStack; Loki itself still doesn't natively ingest OTLP. Locally,
pushing OTLP logs straight from the OpenTelemetry Java agent's logback/log4j2
instrumentation into VictoriaLogs is validated and simple, but it is a
different data path than production and should be documented as a pragmatic
local-dev shortcut, not a faithful mirror of the cluster's log-collection
mechanism.
* *Trace/log correlation*: validated to work *automatically* via the OTel Java
agent's logback-appender instrumentation
({{-Dotel.instrumentation.logback-appender.enabled=true}}) -- every ingested
log line carries real {{trace_id}}/{{span_id}} with zero app-side MDC
configuration. The {{camel-mdc}}-based approach documented in
{{camel-opentelemetry2}}'s docs is not required for this path.
h3. Empirical validation (PoC results)
A working proof-of-concept was built and torn down; all 4 signals (metrics,
traces, logs, and trace/log correlation) were confirmed working end-to-end with
real data from a real {{camel-observability-services}}-enabled Camel route, and
Perses accepted all 3 datasource definitions with no schema errors. Key
findings to carry into implementation:
* VictoriaTraces' OTLP gRPC receiver is *disabled by default* -- must pass
{{-otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false}}.
* VictoriaTraces' Jaeger-compatible API is served at
{{/select/jaeger/api/...}}, not the bare {{/api/...}} its docs imply.
* *Port collision*: Perses defaults to port 8080, the same default port used by
Camel/Spring Boot/Quarkus apps. Fix is to remap Perses, not the app --
confirmed via {{-web.listen-address :3000}} (also verified live), keeping the
app on its normal default port. Suggest defaulting the test-infra service to
port 3000 for Perses, matching the Grafana/otel-lgtm convention devs may
already be used to.
* Minimal working docker commands (see also PoC files, not retained beyond this
session):
{code}
docker network create camel-obs
docker run -d --name victoria-traces --network camel-obs -p 8428:8428 -p
4317:4317 \
victoriametrics/victoria-traces:latest -httpListenAddr=:8428
-otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
docker run -d --name victoria-logs --network camel-obs -p 9428:9428 \
victoriametrics/victoria-logs:latest -httpListenAddr=:9428
docker run -d --name prometheus --network camel-obs -p 9090:9090 \
-v "$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml:ro"
--add-host=host.docker.internal:host-gateway \
prom/prometheus:latest
docker run -d --name perses --network camel-obs -p 3000:3000 \
persesdev/perses:latest -web.listen-address :3000
{code}
* A minimal {{camel-observability-services}}-enabled JBang app, run via {{camel
run}} (a bare {{RouteBuilder}} has no {{main}}, so plain {{jbang run}} does not
work), with the OpenTelemetry Java agent supplying traces+logs export,
confirmed all of: Prometheus scraping {{/observe/metrics}}, a real Camel-level
{{timer}} span in VictoriaTraces, and OTLP log ingestion into VictoriaLogs
correlated back to that trace.
h3. Scope
* New test-infra module/service(s) following existing test-infra patterns (e.g.
{{*InfraService}}, {{*LocalContainerService}}, {{*ServiceFactory}}), composing
the *4 separate containers* above -- there is no single prebuilt image bundling
Prometheus + VictoriaTraces + VictoriaLogs + Perses the way
{{grafana/otel-lgtm}} bundles its stack.
* Wire the new service into the {{camel-jbang-core}} {{infra}} command so
{{camel infra run observability}} works like other {{camel infra run
<service>}} commands, including supplying the Prometheus scrape-target wiring
against the running app.
* Decide the relationship to the existing {{camel-test-infra-jaeger}} module
(coexist vs. supersede) as part of implementation.
* Out of scope for this ticket: changes to {{camel-observability-services}}'
default export behavior itself (e.g. any future default-on MDC/log-correlation
improvement is a separate, runtime-behavior-changing ticket), and cluster-side
OpenShift/Kubernetes tooling (traits, {{camel kubernetes doctor}},
PodMonitor/PrometheusRule generation).
h3. Open items
* Perses' live query-through-datasource-proxy path was not confirmed during the
PoC (datasource definitions were accepted, but querying data through Perses'
proxy returned errors under the guessed path conventions) -- needs checking
against Perses source/docs during implementation.
was:
h3. Problem / Motivation
Camel's local dev loop ({{camel run --dev}}) and the Kubernetes export path
already wire up {{camel-observability-services}} (Micrometer/Prometheus
metrics, health checks, OpenTelemetry tracing) when targeting
Kubernetes/OpenShift. However, there is no local backend to receive or
visualize that telemetry: {{camel infra run}} can start backing services like
Postgres or Artemis, but nothing analogous exists for observability.
As a result, developers only find out whether traces, metrics, and logs
actually work correctly (correct span correlation, meaningful dashboards, sane
metric cardinality, etc.) once the application is deployed to a real OpenShift
cluster, which typically provides the Red Hat build of OpenTelemetry, the Tempo
Operator, Loki/OpenShift Logging, and the Cluster Observability Operator
(Grafana). Telemetry problems surface late, in a shared cluster environment,
rather than at dev/E2E time on a laptop or in CI.
h3. Proposed Solution
Add a new {{camel-test-infra}} service, invocable as {{camel infra run
observability}} (same UX as the existing {{camel infra run postgres}} / {{camel
infra run artemis}} services), that starts a local all-in-one observability
stack via Testcontainers -- for example the {{grafana/otel-lgtm}} image, which
bundles an OTel collector plus Prometheus, Tempo, Loki, and Grafana.
The service should be wired to the same OTLP endpoint / ports / paths that
{{camel-observability-services}} already exports (metrics, health, tracing), so
that the dashboards, traces, and logs a developer sees locally when running
{{camel run --dev}} against this service are the same shape as what they will
see after deploying to OpenShift. This closes the local-vs-production
observability gap and gives a fast, zero-cluster feedback loop for validating
telemetry during the inner dev loop and in E2E test workflows.
h3. Scope
* New test-infra module/service following the existing test-infra patterns
(e.g. an {{*InfraService}}, {{*LocalContainerService}}, and {{*ServiceFactory}}
for the {{grafana/otel-lgtm}} container).
* Wire the new service into the {{camel-jbang-core}} {{infra}} command so
{{camel infra run observability}} works like other {{camel infra run
<service>}} commands.
* Out of scope for this ticket: any changes to {{camel-observability-services}}
itself, and cluster-side OpenShift/Kubernetes tooling (traits, {{camel
kubernetes doctor}}, PodMonitor/PrometheusRule generation) -- those are
separate, larger efforts.
Summary: camel-test-infra: add an "observability" service (Prometheus +
VictoriaTraces + VictoriaLogs + Perses, Apache-2.0 stack) for local dev and
E2E, mirroring OpenShift's observability stack (was: camel-test-infra: add an
"observability" service (Grafana/Tempo/Loki OTel-LGTM stack) for local dev and
E2E, mirroring OpenShift's observability stack)
> camel-test-infra: add an "observability" service (Prometheus + VictoriaTraces
> + VictoriaLogs + Perses, Apache-2.0 stack) for local dev and E2E, mirroring
> OpenShift's observability stack
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24051
> URL: https://issues.apache.org/jira/browse/CAMEL-24051
> Project: Camel
> Issue Type: New Feature
> Components: test-infra
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
>
> h3. Problem / Motivation
> Camel's local dev loop ({{camel run --dev}}) and the Kubernetes export path
> already wire up {{camel-observability-services}} (Micrometer/Prometheus
> metrics, health checks, OpenTelemetry tracing via the bundled
> {{camel-opentelemetry2}}) when targeting Kubernetes/OpenShift. However, there
> is no local backend to receive or visualize that telemetry: {{camel infra
> run}} can start backing services like Postgres or Artemis, but nothing
> analogous exists for observability.
> As a result, developers only find out whether traces, metrics, and logs
> actually work correctly (correct span correlation, meaningful dashboards,
> sane metric cardinality, etc.) once the application is deployed to a real
> OpenShift cluster, which typically provides the Red Hat build of
> OpenTelemetry, the Tempo Operator, Loki/OpenShift Logging, and the Cluster
> Observability Operator (Grafana). Telemetry problems surface late, in a
> shared cluster environment, rather than at dev/E2E time on a laptop or in CI.
> h3. Licensing note (why not grafana/otel-lgtm)
> The obvious candidate for this is Grafana's {{grafana/otel-lgtm}} all-in-one
> image. It was *ruled out*: Grafana, Loki, and Tempo were relicensed from
> Apache 2.0 to AGPLv3 in 2021. AGPLv3 is Category X on the ASF's third-party
> license policy, so an ASF project cannot depend on it even as opt-in test/dev
> tooling without a real [email protected] sign-off, and Red Hat's OSPO
> treats AGPL as off-limits even more strictly, including for test-only tooling
> in downstream builds. Only Prometheus and the OpenTelemetry Collector in that
> bundle are actually Apache 2.0.
> h3. Proposed Solution -- Apache-2.0-only stack (validated with a working PoC,
> see below)
> Use an equivalent stack built entirely from Apache-2.0-licensed components:
> * *Metrics*: Prometheus (unchanged -- already Apache 2.0, already the
> {{camel-observability-services}} default via {{/observe/metrics}}).
> * *Traces*: https://github.com/VictoriaMetrics/VictoriaTraces[VictoriaTraces]
> (Apache 2.0) -- ingests OTLP natively and exposes a Jaeger Query
> Service-compatible JSON API. Functionally a drop-in swap for Tempo.
> * *Logs*: https://github.com/VictoriaMetrics/VictoriaLogs[VictoriaLogs]
> (Apache 2.0) -- LogsQL query language, accepts OTLP logs directly via
> {{/insert/opentelemetry/v1/logs}}. Swap for Loki.
> * *Dashboard*: https://perses.dev[Perses] (Apache 2.0, CNCF sandbox project)
> -- dashboard-as-code tool backed by Amadeus/SAP/Red Hat, and notably *already
> powers OpenShift's own new traces UI*, which makes it a much safer downstream
> bet than Grafana. Its default plugin set already ships Prometheus, Jaeger,
> and VictoriaLogs datasource kinds with no extra plugin install.
> Adding this also revisits the fate of the existing
> {{camel-test-infra-jaeger}} module (added in CAMEL-23641): since
> VictoriaTraces exposes a Jaeger-compatible query API,
> {{camel-test-infra-jaeger}} does *not* need to be deprecated as originally
> floated -- it can keep serving its current role, or the new service can point
> at it directly, instead of being replaced outright. This should be decided
> during implementation, not assumed here.
> h3. Signal-level design rationale (matching real K8s/OCP behavior where it
> matters)
> * *Traces -- push*: matches OCP's own model (Red Hat build of OpenTelemetry
> pushes OTLP spans to a Collector). No divergence between local and cluster
> behavior.
> * *Metrics -- pull*: OCP's real default is Prometheus scraping via
> {{ServiceMonitor}}/{{PodMonitor}}, not app-side push.
> {{camel-observability-services}}' existing Prometheus-pull default already
> matches this -- the local service's job is to supply a scrape target pointed
> at the running app, mirroring what a {{ServiceMonitor}} does on-cluster,
> *not* to switch metrics to OTLP push.
> * *Logs -- local shortcut, not literal OCP parity*: real OpenShift logging
> does not have the app push logs at all -- Vector tails container
> stdout/stderr and ships to LokiStack; Loki itself still doesn't natively
> ingest OTLP. Locally, pushing OTLP logs straight from the OpenTelemetry Java
> agent's logback/log4j2 instrumentation into VictoriaLogs is validated and
> simple, but it is a different data path than production and should be
> documented as a pragmatic local-dev shortcut, not a faithful mirror of the
> cluster's log-collection mechanism.
> * *Trace/log correlation*: validated to work *automatically* via the OTel
> Java agent's logback-appender instrumentation
> ({{-Dotel.instrumentation.logback-appender.enabled=true}}) -- every ingested
> log line carries real {{trace_id}}/{{span_id}} with zero app-side MDC
> configuration. The {{camel-mdc}}-based approach documented in
> {{camel-opentelemetry2}}'s docs is not required for this path.
> h3. Empirical validation (PoC results)
> A working proof-of-concept was built and torn down; all 4 signals (metrics,
> traces, logs, and trace/log correlation) were confirmed working end-to-end
> with real data from a real {{camel-observability-services}}-enabled Camel
> route, and Perses accepted all 3 datasource definitions with no schema
> errors. Key findings to carry into implementation:
> * VictoriaTraces' OTLP gRPC receiver is *disabled by default* -- must pass
> {{-otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false}}.
> * VictoriaTraces' Jaeger-compatible API is served at
> {{/select/jaeger/api/...}}, not the bare {{/api/...}} its docs imply.
> * *Port collision*: Perses defaults to port 8080, the same default port used
> by Camel/Spring Boot/Quarkus apps. Fix is to remap Perses, not the app --
> confirmed via {{-web.listen-address :3000}} (also verified live), keeping the
> app on its normal default port. Suggest defaulting the test-infra service to
> port 3000 for Perses, matching the Grafana/otel-lgtm convention devs may
> already be used to.
> * Minimal working docker commands (see also PoC files, not retained beyond
> this session):
> {code}
> docker network create camel-obs
> docker run -d --name victoria-traces --network camel-obs -p 8428:8428 -p
> 4317:4317 \
> victoriametrics/victoria-traces:latest -httpListenAddr=:8428
> -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
> docker run -d --name victoria-logs --network camel-obs -p 9428:9428 \
> victoriametrics/victoria-logs:latest -httpListenAddr=:9428
> docker run -d --name prometheus --network camel-obs -p 9090:9090 \
> -v "$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml:ro"
> --add-host=host.docker.internal:host-gateway \
> prom/prometheus:latest
> docker run -d --name perses --network camel-obs -p 3000:3000 \
> persesdev/perses:latest -web.listen-address :3000
> {code}
> * A minimal {{camel-observability-services}}-enabled JBang app, run via
> {{camel run}} (a bare {{RouteBuilder}} has no {{main}}, so plain {{jbang
> run}} does not work), with the OpenTelemetry Java agent supplying traces+logs
> export, confirmed all of: Prometheus scraping {{/observe/metrics}}, a real
> Camel-level {{timer}} span in VictoriaTraces, and OTLP log ingestion into
> VictoriaLogs correlated back to that trace.
> h3. Scope
> * New test-infra module/service(s) following existing test-infra patterns
> (e.g. {{*InfraService}}, {{*LocalContainerService}}, {{*ServiceFactory}}),
> composing the *4 separate containers* above -- there is no single prebuilt
> image bundling Prometheus + VictoriaTraces + VictoriaLogs + Perses the way
> {{grafana/otel-lgtm}} bundles its stack.
> * Wire the new service into the {{camel-jbang-core}} {{infra}} command so
> {{camel infra run observability}} works like other {{camel infra run
> <service>}} commands, including supplying the Prometheus scrape-target wiring
> against the running app.
> * Decide the relationship to the existing {{camel-test-infra-jaeger}} module
> (coexist vs. supersede) as part of implementation.
> * Out of scope for this ticket: changes to {{camel-observability-services}}'
> default export behavior itself (e.g. any future default-on
> MDC/log-correlation improvement is a separate, runtime-behavior-changing
> ticket), and cluster-side OpenShift/Kubernetes tooling (traits, {{camel
> kubernetes doctor}}, PodMonitor/PrometheusRule generation).
> h3. Open items
> * Perses' live query-through-datasource-proxy path was not confirmed during
> the PoC (datasource definitions were accepted, but querying data through
> Perses' proxy returned errors under the guessed path conventions) -- needs
> checking against Perses source/docs during implementation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
