Hey Fabian, Thanks for the reply! I did check out the it_exemplars_otel_agent and followed the instructions here
https://github.com/prometheus/client_java/blob/master/OTEL_EXEMPLARS.md for getting that up and working. I used Otel 1.9.1 to match my version and was able to see exemplars with this test code. $ java -Dotel.traces.exporter=logging -Dotel.metrics.exporter=none -javaagent:./opentelemetry-javaagent.jar -jar ./exa mple-spring-boot-app.jar $ curl -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' http://localhost:8080/metrics | grep sp an_id requests_total{path="/hello"} 3.0 # {span_id="4f836ba5e1b35cc2",trace_id="0d02651c1faf5b6aeb9a2cabdcdc0009"} 1.0 1641568523.154 requests_total{path="/god-of-fire"} 3.0 # {span_id="d00084d7529b0881",trace_id="0d02651c1faf5b6aeb9a2cabdcdc0009"} 1.0 1641568523.278 So the test code definitely works. Now I'm trying to understand what's different in my code in GitHub so that I can reproduce these results. The only real things that stood out to me are: * The hotspot default metrics were missing in my code -- I've added these, but this didn't fix exemplars * I'm manually instrumenting exemplars in all but one Counter * I'm using the Scheduler for repeated tasks rather than a REST-like app. But I'm out of ideas about what to try next here and I'm looking for some assistance as to what the issue is with my example code. Again, that lives here: https://github.com/jjneely/java-exemplar-example Jack Neely On Thu, Jan 6, 2022, at 16:13, Fabian Stäber wrote: > Hi Jack, > > You can find a working example in the it_exemplars_otel_agent > integration test. The test runs the ExampleSpringBootApp in a Docker > container with the OpenTelemetry agent attached. > ExemplarsOpenTelemetryAgentIT verifies that Counter and Histogram > metrics have Exemplars if they were observed in the context of a > distributed trace. > > Fabian > > On Thu, Jan 6, 2022 at 9:11 PM Jack Neely <[email protected]> wrote: >> Friends, >> >> I wrote about this issue some months ago when I was trying to put together a >> company wide demo and running out of time. My apologies for not following >> through. >> >> I'm attempting to write a simple Java Spring framework app that serves as a >> working example of the metric and logging practices we'd like to follow as >> our standards. I've been trying for a while now to get exemplar support >> working to demo how to pivot from metrics to traces, but I've never been >> successful in getting exemplars to appear. >> >> I've copied the example code to GitHub. >> >> https://github.com/jjneely/java-exemplar-example >> >> This is a super simple Gradle / Spring Boot app and running "./gradlew >> bootrun" should get it up and running. Log output will indicate if exemplar >> support is enabled and it should be as it runs with the OpenTelemetry Agent >> 1.9.1. However, using curl against the metric endpoint does not show >> exemplars and having Prometheus scrape the endpoint does not show exemplars. >> >> $ curl -H 'Accept: application/openmetrics-text; version=1.0.0; >> charset=utf-8' http://localhost:8081/ >> >> I've manually added exemplar values in the code, yet no luck. I feel like >> I'm missing something simple here. I'm hoping this issue is reproducible >> and someone can help me figure out what I've missed here. >> >> I've also sent a note in the CNCF Slack: >> >> https://cloud-native.slack.com/archives/C167KFM6C/p1641494335010700 >> >> Thanks for the help! Now that I'm not under a time crunch I promise to get >> back to folks. >> >> Jack Neely >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:prometheus-developers%[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/73deecb0-da81-4970-ae20-66cc3543f7c1%40www.fastmail.com. -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/361cfa19-a8fc-4666-874e-862fb8f35295%40www.fastmail.com.
