Thank you Brian. Here's my use case:
I use PushGateway to allows Prometheus to scrape a custom component in our service. Here the metric is the number of notifications sent for a given service instance. So every time there is a Push Notification being sent via the service instance, I go ahead and increment the Guage by 1 and send it to PushGateway. Prometheus then scrapes this and sends it to Sysdig. Currently what happens is if the user happened to send 5 notifications to say at 12:58 PM and then does not send any more notification, what I see is 5 is being sent by the Prometheus server even after 12:58 PM and that continues until the next time a notification is being sent by the user and the value changes. So the way I would want it to work is at 12:58 PM it should spike up showing that the value is 5 indicating that there were 5 notifications that were sent and then drop back to 0 until the next value comes in. On Mon, 12 Oct 2020 at 12:47, Brian Candler <[email protected]> wrote: > > Should it not stop relaying the same data to sysdig once it has sent it > successfully? > > No, because it's not the same data. Each scrape creates a new data point: > - The value of this metric was V at time T1 > - The value of this metric was V at time T2 > - The value of this metric was V at time T3 > ... etc > > Those are different pieces of information. > > Pushgateway is not a "relay" as such. It is more of a cache, just storing > the last-written value. > > The model of Prometheus is: > - data is fetched via scraping > - you can have multiple Prometheus instances scraping the same exporters > (e.g. for high availability, or so you can run a development instance on > your laptop) > > Therefore it would not be possible for pushgateway to delete a value once > it has been scraped; that would stop a second Prometheus instance from > seeing the same data. > > But there are other reasons why this is a bad idea. One of them is that > any timeseries which has not had a write within the last 5 minutes is > considered "stale", i.e. it has no current data. If you ask the value of a > metric at a given instant, Prometheus only looks 5 minutes into the past. > So if the value of a metric is V, then you need to record the value > repeatedly to be able to see that its current value is still V. There is a > difference between knowing the value of something is the same as it was > before, and not knowing its value at all. > > > So how do I prevent Prometheus from relaying the same data > > You would need to let the timeseries become stale - i.e. "no data" rather > than "same data as last time". You can't do that with pushgateway: see > https://github.com/prometheus/pushgateway#non-goals > > The solution is to get rid of pushgateway, and write a proper exporter for > your data: one which is scraped directly, and returns a list of all the > current values of a metric. If a metric no longer exists, then you can > stop returning it, and it will become stale. But as I said before, "a > metric no longer exists" is different to "the metric still exists, but > happens to have the same value as last time you saw it". > > If you describe what your metric is and what it represents, people may be > able to help you more. > > If you are only using pushgateway because the source cannot be scraped > directly (e.g. because of firewall restrictions), then there are other > options, e.g. PushProx <https://github.com/prometheus-community/PushProx>. > >> -- > You received this message because you are subscribed to a topic in the > Google Groups "Prometheus Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/prometheus-users/uGYUQhQAdOE/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/81a57343-8f6c-48b6-9e27-9954718936ffo%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/81a57343-8f6c-48b6-9e27-9954718936ffo%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAEdsq%2Byx5LegR3jOnn997W2iN0VpT_45BYukQhs2qaU169u_hA%40mail.gmail.com.
