On Tue, Jan 5, 2021 at 5:36 PM Roman Dodin <[email protected]> wrote:
> Thank you for your comments, Stuart, maybe I expressed myself a bit vague. > Let me be more precise and maybe then it will be easier to answer my > question. > > The metric I am talking about is the number of routes a network element > has in its routing table. This integer is reported to the collector only > when it's changed (i.e. a new route has been added to the table and now the > number of routes is X+1). > The collector receives the number of routes and exposes this metric for > Prometheus to scrape. Prometheus successfully scrapes this metric and > stores in TSDB. > > If no more routes are added to the system for a period T, no metrics will > be available for Prometheus to scrape, but at the same time, in my view, it > is not an event, it is a metric, it is just not periodically > reported, because there is no changes to it. We are reducing the amount of > data to transfer and store, by not sending the data that hasn't changed. > > I am curious, if that makes the original question clearer? Is it not > against the design to use Prometheus for such metrics and with this > particular scraping strategy? > Yes, it's more clear. This data behavior is incompatible with Prometheus due to the way stale data is handled. If a thing exists, it should continue to be exported even if it's not changing. Prometheus deals with this just fine, as it uses compression to store the data. In the case of non-changing values, the storage requirements are extremely trivial, on the order of a few bits. In fact, not repeating is almost no savings, as there is a minimum of 120 samples stored in a 2 hour window. I suggest changing your exporter behavior to always produce the "current state of the world". > > On Tue, Jan 5, 2021 at 5:14 PM Stuart Clark <[email protected]> > wrote: > >> On 05/01/2021 15:35, Roman Dodin wrote: >> >> Hello community, >> I have a telemetry system that sends data into Prometheus when the data >> changes (i.e. its a push-on-event mechanism, not a sample/interval push) >> That means, if a system is stable, then the last reported value can be >> reported quite some time back. >> >> Let's say this metric is called "my_metric", how do I get its last >> reported value without specifying manually the timeframe to look back? >> So far I come up with the following query: >> >> my_metric{label="value"}[100y] >> >> Is this the only way to get the last value of a series, or maybe there is >> another alternative? >> >> Prometheus is designed to be used as a scrape (pull) system where metrics >> are regularly fetched and their values recorded in the TSDB (even if they >> haven't changed). As a result of this and metric that doesn't have a recent >> value is seen as "stale" and isn't returned when doing instant queries. The >> default setting is 5 minutes, which leads to the generally suggested >> maximum scrape interval of 2 minutes (to allow for a single scrape failure >> without the series being marked as stale). >> >> I'm not sure how you are pushing events into Prometheus, but it sounds >> like you are working against the design of the system. Additionally >> Prometheus is a metrics system rather than an event storage system. >> >> For event storage I would use a different database (possibly SQL based or >> a generic timeseries database) which would have no problems with the sort >> of queries you are wanting. >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAFBEvLtEbUhutdYcBc58Zme7tCOT-nVacod2tAWPMuYq954w_Q%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CAFBEvLtEbUhutdYcBc58Zme7tCOT-nVacod2tAWPMuYq954w_Q%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmriYzK1XZAwxg94OFTC%2B_EZ_tqLPmCQZr09YdPVXtp9MA%40mail.gmail.com.
