Ah interesting. I did a quick test with `deriv()` function with wan_inIfOctets and that seems to have fixed the issue. However, according to Prometheus docs, it says it should be used for gauge metrics instead of counter. Any drawbacks to using `deriv()` over `rate()`?
On Tuesday, September 22, 2020 at 2:24:39 PM UTC-4 [email protected] wrote: > This is what I suspected. The custom exporter is producing values that > drop slightly from scrape to scrape, triggering Prometheus into thinking > the counter was reset. > > If you did a deriv() function, you would see a negative traffic value. > > On Tue, Sep 22, 2020, 19:18 Adam <[email protected]> wrote: > >> Here's the result of `resets(wan_ifInOctets{site=~"$site"}[5m])`: >> [image: Screenshot 2020-09-22 131755.png] >> >> On Tuesday, September 22, 2020 at 12:30:00 PM UTC-4 [email protected] >> wrote: >> >>> What if you graph `resets(wan_ifInOctets{site=~"$site"}[5m])`? >>> >>> On Tue, Sep 22, 2020 at 6:03 PM Adam <[email protected]> wrote: >>> >>>> I am trying to troubleshoot the cause of random jumps in the network >>>> activity. I have a custom exporter that reports the current network >>>> activity from WAN routers via SNMP. The data is scraped every two minutes >>>> and this issue only happens on specific routers while others report no >>>> issues. >>>> >>>> Here's a picture to describe the issue I'm facing: >>>> [image: Screenshot 2020-09-22 115806.png] >>>> *This metrics data is the same from Prometheus' end. I used Grafana to >>>> be able to show the issue I'm having.* >>>> >>>> As you can tell, the raw_172.19.187.146 (yellow line) is going at a >>>> smooth increase while raw_172.19.187.146 shows a massive jump for at least >>>> 2-3 minutes. >>>> >>>> Any idea why this issue is happening? Is the query in A row acceptable >>>> to show the network activity every five minutes (since Prometheus scrapes >>>> it every 2 minutes)? >>>> >>>> Thank you and I look forward to your response. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/e43bcf3a-2a20-4ded-b782-1c8d28028643n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/e43bcf3a-2a20-4ded-b782-1c8d28028643n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/e953ad75-a60a-4fa4-aa96-6e3e39366330n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/e953ad75-a60a-4fa4-aa96-6e3e39366330n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/64cf9d93-b5fa-47bf-97ff-cd2085153f40n%40googlegroups.com.
