Yes, this sounds a lot like duplicate sample ingestion or some other non-compliant metrics endpoint.
It would also be helpful to have the prometheus.yml configuration to understand what else is going on. On Sat, Jun 20, 2020 at 10:30 AM Julius Volz <[email protected]> wrote: > When I create a file "metrics" with your metrics from the 2.4 example > block and serve it to Prometheus (using just "python -m SimpleHTTPServer > 12345") and then query for "my_metric" from a 2.18 Prometheus, I don't see > this problem. > > Would it be feasible to share a minimal /metrics endpoint example that > reproduces this behavior in 2.18? > > On Sat, Jun 20, 2020 at 3:55 AM Johny <[email protected]> wrote: > >> I recently upgraded Prometheus version 2.4 to 2.18. It seems there is >> some incompatibility in the api. When I query a metric, I get same value >> across all time series in 2.18. >> >> e.g. >> query: my_metric >> >> 2.4 api- >> Element Value >> my_metric{l1="a1",l2="b1",l3="x-y-z1"} 3434 >> my_metric{l1="a2",l2="b2",l3="x-y-z2"} 3.433 >> my_metric{l1="a3",l2="b3",l3="x-y-z3"} 94344 >> my_metric{l1="a4",l2="b4",l3="x-y-z4"} 1000 >> >> 2.18 api returns the same value for all time series- >> Element Value >> my_metric{l1="a1",l2="b1",l3="x-y-z1"} 1000 >> my_metric{l1="a2",l2="b2",l3="x-y-z2"} 1000 >> my_metric{l1="a3",l2="b3",l3="x-y-z3"} 1000 >> my_metric{l1="a4",l2="b4",l3="x-y-z4"} 1000 >> >> The same problem occurs when I query across time. If I plot >> count(my_metric) on the graph, I get a flat line in 2.18 whereas 2.4 >> changes over time. >> >> However, when I query a single time series, I don't have this issue in >> 2.18. >> >> Is this a known incompatibility issue between the two versions and does >> it affect storage layer as well? >> How do I solve it? >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/191deffd-faf7-4370-a907-15863e84f64ao%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/191deffd-faf7-4370-a907-15863e84f64ao%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAObpH5wg9q8S8yYUFs9cP5Ctoir5hqQ%3DVo4XMJt05WW6iKXX-w%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CAObpH5wg9q8S8yYUFs9cP5Ctoir5hqQ%3DVo4XMJt05WW6iKXX-w%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpHK_cSYan_hTqaf8%2B2ZK03wy4QBD7ryzm3HR9-5eKvyA%40mail.gmail.com.
