https://github.com/prometheus/pushgateway/issues/371
I have read #19 <https://github.com/prometheus/pushgateway/issues/19> discussion and many others on the topic of TTL in pushgateway. This proposal is slightly different: [optionally] remove the metric when it's been read (probably by Prometheus) My use case and seemingly many other people's call for it. As it is, I have to jump through hoops to works around the issue of metric still showing as current in Prometheus long after it's been pushed to PG Copying the response from beorn7 below in github and my response to that: 1. I am NOT doing event processing. 2. We manage a large Hadoop cluster (6000 nodes) and Hadoop services are being monitored by PUSHing metrics to a different system. 3. We want to send these metrics to Prometheus 4. The format of these metrics is Not in Prometheus format, so cannot be scraped, AND we are not setup for such scraping anyway, we are setup for PUSH. 5. I would rather not enter a discussion about merits of Push vs Pull and how a system could be re-architected from scratch. 6. Pushgateway satisfies the use case, however a metric needs to be scraped once and deleted, hence the proposal. 7. Understood the point about HA, that's why I propose OPTIONAL removal - let the user decide. *beorn7 <https://github.com/beorn7> *commented 16 hours ago <https://github.com/prometheus/pushgateway/issues/371#issuecomment-719531146> It's a fundamental property of Prometheus metrics endpoints that a scrape doesn't change the state of the metrics. For example, in a Prometheus HA setup, two identically configured Prometheus servers scrape the same metrics endpoint. The one server in such an HA pair would trigger the removal of the metric in your proposal, and the other server would then not see it anymore. If you need a "delete after scraping once" behavior, that's a strong hint that you are actually doing event processing, for which Prometheus is not the right choice. If you have more questions about your use case and how to implement (or perhaps better not to implement) it with Prometheus, I recommend the prometheus-users mailing list <https://groups.google.com/forum/#!forum/prometheus-users>. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4ecec86f-3a4b-4bc3-99e2-0ea55898f69an%40googlegroups.com.
