Yes deriv() doesn't handle real counter resets and produces negative values.
It's not fixed, you're just ignoring the fact that it's almost certainly producing negative rates. Something is funky about your timestamps. It looks like two different scrape jobs are being mixed into a single series. On Tue, Sep 22, 2020 at 8:59 PM Adam <[email protected]> wrote: > Ah interesting. I did a quick test with `deriv()` function with > wan_inIfOctets and that seems to have fixed the issue. However, according > to Prometheus docs, it says it should be used for gauge metrics instead of > counter. Any drawbacks to using `deriv()` over `rate()`? > > On Tuesday, September 22, 2020 at 2:24:39 PM UTC-4 [email protected] wrote: > >> This is what I suspected. The custom exporter is producing values that >> drop slightly from scrape to scrape, triggering Prometheus into thinking >> the counter was reset. >> >> If you did a deriv() function, you would see a negative traffic value. >> >> On Tue, Sep 22, 2020, 19:18 Adam <[email protected]> wrote: >> >>> Here's the result of `resets(wan_ifInOctets{site=~"$site"}[5m])`: >>> [image: Screenshot 2020-09-22 131755.png] >>> >>> On Tuesday, September 22, 2020 at 12:30:00 PM UTC-4 [email protected] >>> wrote: >>> >>>> What if you graph `resets(wan_ifInOctets{site=~"$site"}[5m])`? >>>> >>>> On Tue, Sep 22, 2020 at 6:03 PM Adam <[email protected]> wrote: >>>> >>>>> I am trying to troubleshoot the cause of random jumps in the network >>>>> activity. I have a custom exporter that reports the current network >>>>> activity from WAN routers via SNMP. The data is scraped every two minutes >>>>> and this issue only happens on specific routers while others report no >>>>> issues. >>>>> >>>>> Here's a picture to describe the issue I'm facing: >>>>> [image: Screenshot 2020-09-22 115806.png] >>>>> *This metrics data is the same from Prometheus' end. I used Grafana to >>>>> be able to show the issue I'm having.* >>>>> >>>>> As you can tell, the raw_172.19.187.146 (yellow line) is going at a >>>>> smooth increase while raw_172.19.187.146 shows a massive jump for at least >>>>> 2-3 minutes. >>>>> >>>>> Any idea why this issue is happening? Is the query in A row acceptable >>>>> to show the network activity every five minutes (since Prometheus scrapes >>>>> it every 2 minutes)? >>>>> >>>>> Thank you and I look forward to your response. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/e43bcf3a-2a20-4ded-b782-1c8d28028643n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/prometheus-users/e43bcf3a-2a20-4ded-b782-1c8d28028643n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/e953ad75-a60a-4fa4-aa96-6e3e39366330n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/e953ad75-a60a-4fa4-aa96-6e3e39366330n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/64cf9d93-b5fa-47bf-97ff-cd2085153f40n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/64cf9d93-b5fa-47bf-97ff-cd2085153f40n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoEnkgNniTsHSwSE5VXomSQJvv%2B7WAZxaZN9%2Bm7yJWrkg%40mail.gmail.com.
