I am building a demo application to show off Prometheus Exemplars and
OpenTelemetry Tracing abilities with Java / Spring applications. However,
I cannot get exemplars to appear in the metrics HTTP endpoint. I'm using
the following curl command and I get metrics in the OpenMetrics format, but
no exemplars.
$ curl -H 'Accept: application/openmetrics-text; version=1.0.0;
charset=utf-8' http://localhost:8081/metrics
I've got the Prometheus client_java 0.12.0 loaded and I'm using that to
directly instrument. In build.gradle:
dependencies {
...
implementation 'io.prometheus:simpleclient:0.12.0'
implementation 'io.prometheus:simpleclient_hotspot:0.12.0'
implementation 'io.prometheus:simpleclient_httpserver:0.12.0'
implementation 'io.prometheus:simpleclient_tracer_otel:0.12.0'
implementation 'io.prometheus:simpleclient_tracer_otel_agent:0.12.0'
implementation('io.opentelemetry:opentelemetry-api:1.0.0')
implementation('io.opentelemetry:opentelemetry-extension-annotations:1.0.0')
}
I setup my Prometheus metrics in the class constructor:
prometheusErrors = io.prometheus.client.Counter.build()
.namespace("custommetricsdemo")
.name("errors")
.help("Test Prometheus Client Library errors counter")
.withExemplars()
.register();
prometheusTimer = io.prometheus.client.Summary.build()
.namespace("custommetricsdemo")
.name("latency_timer")
.help("Test Prometheus Client Library latency summary")
.quantile(0.5, 0.01)
.quantile(0.95, 0.01)
.register();
prometheusHistogram = io.prometheus.client.Histogram.build()
.namespace("custommetricsdemo")
.name("histogram")
.help("Test Prometheus Client Library latency histogram")
.withExemplars()
.register();
try {
io.prometheus.client.exporter.HTTPServer server
= new io.prometheus.client.exporter.HTTPServer(8081);
} catch (Exception e) {
logger.error("Failed to setup Prometheus HTTP server", e);
}
This code runs a ScheduledTask at a random interval which sleeps for a
random time. Thanks to running with the OpenTelemetry Agent, I can see
trace_id and span_id in the logback logs. That part works well. I can
also query the span IDs in code:
logger.info("{}, {}", span.getSpanContext().getSpanId(),
span.getSpanContext().getTraceId());
I could not get the auto-exemplars to work, so I began to observe them in
code:
prometheusErrors.incWithExemplar("span_id",
span.getSpanContext().getSpanId(), "trace_id",
span.getSpanContext().getTraceId());
I also tested with different exemplar tag names:
prometheusHistogram.observeWithExemplar(sw.getTotalTimeSeconds(),
"span_foo", span.getSpanContext().getSpanId(), "trace_bar",
span.getSpanContext().getTraceId());
However, no exemplars ever appear in the output.
$ curl -H 'Accept: application/openmetrics-text; version=1.0.0;
charset=utf-8' http://localhost:8081/metrics
# TYPE custommetricsdemo_histogram histogram
# HELP custommetricsdemo_histogram Test Prometheus Client Library latency
histogram
custommetricsdemo_histogram_bucket{le="0.005"} 3.0
custommetricsdemo_histogram_bucket{le="0.01"} 3.0
custommetricsdemo_histogram_bucket{le="0.025"} 3.0
custommetricsdemo_histogram_bucket{le="0.05"} 4.0
custommetricsdemo_histogram_bucket{le="0.075"} 4.0
custommetricsdemo_histogram_bucket{le="0.1"} 4.0
custommetricsdemo_histogram_bucket{le="0.25"} 6.0
custommetricsdemo_histogram_bucket{le="0.5"} 9.0
custommetricsdemo_histogram_bucket{le="0.75"} 11.0
custommetricsdemo_histogram_bucket{le="1.0"} 11.0
custommetricsdemo_histogram_bucket{le="2.5"} 11.0
custommetricsdemo_histogram_bucket{le="5.0"} 11.0
custommetricsdemo_histogram_bucket{le="7.5"} 11.0
custommetricsdemo_histogram_bucket{le="10.0"} 11.0
custommetricsdemo_histogram_bucket{le="+Inf"} 11.0
custommetricsdemo_histogram_count 11.0
custommetricsdemo_histogram_sum 2.732210076
custommetricsdemo_histogram_created 1.632231018649E9
What am I missing here? Is something interfering with the Spring Boot
libraries perhaps?
Jack Neely
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/7a09c0bc-a002-4643-a8fa-8cb6a24db87dn%40googlegroups.com.
