On 20 Jun 08:31, Johny wrote: > If it is non-compliant endpoint, the problem should appear in both > versions, isn't it? It is effecting more than one series. The set up is in > corporate org so I cannot expose end points publicly. > > I have an prometheus front end instance that remote reads from multiple > prometheus backends. the time series is sharded across multiple backends. > The results are also inconsistent in 2.18.1. Sometimes I get fewer time > series back but what is consistent is the last data point is duplicated on > all time series. Just switching front end to 2.4 with same configuration > file fixes the problem.
Are you using remote read? A bug was fixed in 2.18.2. > > > > On Saturday, June 20, 2020 at 4:43:32 AM UTC-4, Ben Kochie wrote: > > > > Yes, this sounds a lot like duplicate sample ingestion or some other > > non-compliant metrics endpoint. > > > > It would also be helpful to have the prometheus.yml configuration to > > understand what else is going on. > > > > On Sat, Jun 20, 2020 at 10:30 AM Julius Volz <[email protected] > > <javascript:>> wrote: > > > >> When I create a file "metrics" with your metrics from the 2.4 example > >> block and serve it to Prometheus (using just "python -m SimpleHTTPServer > >> 12345") and then query for "my_metric" from a 2.18 Prometheus, I don't see > >> this problem. > >> > >> Would it be feasible to share a minimal /metrics endpoint example that > >> reproduces this behavior in 2.18? > >> > >> On Sat, Jun 20, 2020 at 3:55 AM Johny <[email protected] <javascript:>> > >> wrote: > >> > >>> I recently upgraded Prometheus version 2.4 to 2.18. It seems there is > >>> some incompatibility in the api. When I query a metric, I get same value > >>> across all time series in 2.18. > >>> > >>> e.g. > >>> query: my_metric > >>> > >>> 2.4 api- > >>> Element Value > >>> my_metric{l1="a1",l2="b1",l3="x-y-z1"} 3434 > >>> my_metric{l1="a2",l2="b2",l3="x-y-z2"} 3.433 > >>> my_metric{l1="a3",l2="b3",l3="x-y-z3"} 94344 > >>> my_metric{l1="a4",l2="b4",l3="x-y-z4"} 1000 > >>> > >>> 2.18 api returns the same value for all time series- > >>> Element Value > >>> my_metric{l1="a1",l2="b1",l3="x-y-z1"} 1000 > >>> my_metric{l1="a2",l2="b2",l3="x-y-z2"} 1000 > >>> my_metric{l1="a3",l2="b3",l3="x-y-z3"} 1000 > >>> my_metric{l1="a4",l2="b4",l3="x-y-z4"} 1000 > >>> > >>> The same problem occurs when I query across time. If I plot > >>> count(my_metric) on the graph, I get a flat line in 2.18 whereas 2.4 > >>> changes over time. > >>> > >>> However, when I query a single time series, I don't have this issue in > >>> 2.18. > >>> > >>> Is this a known incompatibility issue between the two versions and does > >>> it affect storage layer as well? > >>> How do I solve it? > >>> > >>> > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > >>> Groups "Prometheus Users" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > >>> an email to [email protected] <javascript:>. > >>> To view this discussion on the web visit > >>> https://groups.google.com/d/msgid/prometheus-users/191deffd-faf7-4370-a907-15863e84f64ao%40googlegroups.com > >>> > >>> <https://groups.google.com/d/msgid/prometheus-users/191deffd-faf7-4370-a907-15863e84f64ao%40googlegroups.com?utm_medium=email&utm_source=footer> > >>> . > >>> > >> > >> > >> -- > >> Julius Volz > >> PromLabs - promlabs.com > >> > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Prometheus Users" group. > >> To unsubscribe from this group and stop receiving emails from it, send an > >> email to [email protected] <javascript:>. > >> To view this discussion on the web visit > >> https://groups.google.com/d/msgid/prometheus-users/CAObpH5wg9q8S8yYUFs9cP5Ctoir5hqQ%3DVo4XMJt05WW6iKXX-w%40mail.gmail.com > >> > >> <https://groups.google.com/d/msgid/prometheus-users/CAObpH5wg9q8S8yYUFs9cP5Ctoir5hqQ%3DVo4XMJt05WW6iKXX-w%40mail.gmail.com?utm_medium=email&utm_source=footer> > >> . > >> > > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/5f49b5ab-d28e-4574-81fd-3a300cbb2c70o%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20200620153433.GA384040%40oxygen.
