As Ben says: it seems that data points are being received in the wrong order, or multiple scrapes getting intermixed, such that the counter sometimes goes down..
It would have been easier if you'd done a copy-paste rather than pasting a screen image, but I can see the timestamps going like this: ...5372 ...5376 ...5492 ...5496 ...5612 ...5616 Notice the intervals of 4 seconds, 116 seconds, 4 seconds, 116 seconds etc. It looks like you're scraping the same target twice, with both scrapes at 120 second intervals, but merging the results into the same timeseries. This is clearly broken. I suspect that either: 1. you've dropped some labels in metric relabelling that shouldn't have been dropped 2. you've changed some labels which should be distinct (e.g. you've overwritten the "job" label in relabelling, so that two different scrape jobs get the same "job" label) 3. you're federating the same timeseries from two different prometheus servers, but forgot to set external_labels to keep the two timeseries distinct 4. you've listed the same target more than once in a targets file (but without applying additional labels to distinguish the scrapes) 5. misuse of "honor_labels: true" 6. something else I haven't thought of You'll need to check your scrape configs to work out how multiple scrapes for the same target could end up in the same timeseries (i.e. with exactly the same label set) -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f1523a95-130a-488d-ae2f-97c19ad13eabo%40googlegroups.com.
