Hi Brian/Stuart: The prometheus implementation has around 2000+JVMs and uses two approaches (JMX exporter or SpringBoot Actuator/Micrometer). The implementation is up to the JVM owner.
JMX exporter exposes metrics at http://host:port/metrics whereas Micrometer exposes them at http://host:port/actuator/metrics. Given we don't know where metrics might be exposed, we have both of them mentioned in prometheus.yaml. This works fine, given we expect metrics to only come from one source. However, we have found cases where they can come through both http://host:port/metrics and http://host:port/actuator/metrics. For these cases, there are duplicate metrics stored (it is redundant data) and also gets double counted in dashboards. It might be quite possible there are incorrect implementations where JVM is configured to use both JMX exporter and Actuator/Micrometer. Thinking of a way of how we can stop storing the duplicates as we find and correct the implementations. The targets to be scrapped are stored in json files which are referred to by prometheus.yaml. We will be able to help with examples of json files and prometheus.yaml if required. Let us know. Thanks, Naveen On Mon, Jan 23, 2023 at 9:08 PM Brian Candler <b.cand...@pobox.com> wrote: > Can you give an specific example of "same metrics are published on two > different endpoints" ? > > You might mean: > - two different metric names > - the same metric name, but different labels > > And it might be that you're scraping the same target twice, or you're > scraping one target but that target is (for some reason) returning > duplicates in the scrape results. Or you might have a more complex > scenario, e.g. multiple prometheus servers scraping for redundancy, and > then you're combining the results together somehow. > > > Is it possible to pick one endpoint and discard the other while writing > a PromQL query ? > > Sure. Just filter in the PromQL query. For example, if you have > > foo{aaa="bbb",ccc="ddd"} 123.0 > foo{aaa="bbb",ccc="fff"} 123.0 > > and you consider the one with ccc="fff" to be a "duplicate" metric, then > > foo{ccc!="fff"} > > might be what you want. > > Otherwise, you can avoid ingesting the duplicate metrics: > - by not scraping the second set in the first place > - if they all come from the same scrape, then using metric_relabel_configs > to drop the metrics that you don't want to keep > > On Monday, 23 January 2023 at 14:40:40 UTC kishore...@gmail.com wrote: > >> Hi, >> We have a situation where same metrics are published on two different >> endpoints. Is it possible to pick one endpoint and discard the other while >> writing a PromQL query ? >> Is it possible to configure Prometheus to collect metrics from only one >> endpoint? >> >> / Kishore >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAJnTqx6rG-WYadmnqA6beVCm5rQrnJGKNbB_qR_ATv273rt7ww%40mail.gmail.com.
