There's a very good chance it's the federate configuration that my noob self set a while back. After some research of federation, I realized I didn't even need it at all so I removed the whole federation configuration and reverted the recording rule from deriv() back to rate().
Hopefully, that should fix the issue. On Wednesday, September 23, 2020 at 3:57:12 AM UTC-4 [email protected] wrote: > As Ben says: it seems that data points are being received in the wrong > order, or multiple scrapes getting intermixed, such that the counter > sometimes goes down.. > > It would have been easier if you'd done a copy-paste rather than pasting a > screen image, but I can see the timestamps going like this: > > ...5372 > ...5376 > ...5492 > ...5496 > ...5612 > ...5616 > > Notice the intervals of 4 seconds, 116 seconds, 4 seconds, 116 seconds etc. > > It looks like you're scraping the same target twice, with both scrapes at > 120 second intervals, but merging the results into the same timeseries. > This is clearly broken. I suspect that either: > > 1. you've dropped some labels in metric relabelling that shouldn't have > been dropped > 2. you've changed some labels which should be distinct (e.g. you've > overwritten the "job" label in relabelling, so that two different scrape > jobs get the same "job" label) > 3. you're federating the same timeseries from two different prometheus > servers, but forgot to set external_labels to keep the two timeseries > distinct > 4. you've listed the same target more than once in a targets file (but > without applying additional labels to distinguish the scrapes) > 5. misuse of "honor_labels: true" > 6. something else I haven't thought of > > You'll need to check your scrape configs to work out how multiple scrapes > for the same target could end up in the same timeseries (i.e. with exactly > the same label set) > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/338e5176-d7fb-463d-aa4a-cc9a108f7fbcn%40googlegroups.com.
