One thing I forgot to ask:
how bad is that query (since for now it does what is needed)
performance-wise?
my_metric{label="value"}[100y]
On Tue, Jan 5, 2021 at 8:57 PM Roman Dodin <[email protected]> wrote:
> Thanks Ben, it is clear now.
> I will check if it makes sense to add a "metric-replay" feature in the
> collector to monotonously export the on_change-type values.
>
>
> On Tue, Jan 5, 2021 at 8:13 PM Ben Kochie <[email protected]> wrote:
>
>>
>>
>> On Tue, Jan 5, 2021 at 5:36 PM Roman Dodin <[email protected]> wrote:
>>
>>> Thank you for your comments, Stuart, maybe I expressed myself a bit
>>> vague.
>>> Let me be more precise and maybe then it will be easier to answer my
>>> question.
>>>
>>> The metric I am talking about is the number of routes a network element
>>> has in its routing table. This integer is reported to the collector only
>>> when it's changed (i.e. a new route has been added to the table and now the
>>> number of routes is X+1).
>>> The collector receives the number of routes and exposes this metric for
>>> Prometheus to scrape. Prometheus successfully scrapes this metric and
>>> stores in TSDB.
>>>
>>> If no more routes are added to the system for a period T, no metrics
>>> will be available for Prometheus to scrape, but at the same time, in my
>>> view, it is not an event, it is a metric, it is just not periodically
>>> reported, because there is no changes to it. We are reducing the amount of
>>> data to transfer and store, by not sending the data that hasn't changed.
>>>
>>> I am curious, if that makes the original question clearer? Is it not
>>> against the design to use Prometheus for such metrics and with this
>>> particular scraping strategy?
>>>
>>
>> Yes, it's more clear. This data behavior is incompatible with Prometheus
>> due to the way stale data is handled. If a thing exists, it should continue
>> to be exported even if it's not changing.
>>
>> Prometheus deals with this just fine, as it uses compression to store the
>> data. In the case of non-changing values, the storage requirements are
>> extremely trivial, on the order of a few bits.
>>
>> In fact, not repeating is almost no savings, as there is a minimum of 120
>> samples stored in a 2 hour window.
>>
>> I suggest changing your exporter behavior to always produce the "current
>> state of the world".
>>
>>
>>>
>>> On Tue, Jan 5, 2021 at 5:14 PM Stuart Clark <[email protected]>
>>> wrote:
>>>
>>>> On 05/01/2021 15:35, Roman Dodin wrote:
>>>>
>>>> Hello community,
>>>> I have a telemetry system that sends data into Prometheus when the data
>>>> changes (i.e. its a push-on-event mechanism, not a sample/interval push)
>>>> That means, if a system is stable, then the last reported value can be
>>>> reported quite some time back.
>>>>
>>>> Let's say this metric is called "my_metric", how do I get its last
>>>> reported value without specifying manually the timeframe to look back?
>>>> So far I come up with the following query:
>>>>
>>>> my_metric{label="value"}[100y]
>>>>
>>>> Is this the only way to get the last value of a series, or maybe there
>>>> is another alternative?
>>>>
>>>> Prometheus is designed to be used as a scrape (pull) system where
>>>> metrics are regularly fetched and their values recorded in the TSDB (even
>>>> if they haven't changed). As a result of this and metric that doesn't have
>>>> a recent value is seen as "stale" and isn't returned when doing instant
>>>> queries. The default setting is 5 minutes, which leads to the generally
>>>> suggested maximum scrape interval of 2 minutes (to allow for a single
>>>> scrape failure without the series being marked as stale).
>>>>
>>>> I'm not sure how you are pushing events into Prometheus, but it sounds
>>>> like you are working against the design of the system. Additionally
>>>> Prometheus is a metrics system rather than an event storage system.
>>>>
>>>> For event storage I would use a different database (possibly SQL based
>>>> or a generic timeseries database) which would have no problems with the
>>>> sort of queries you are wanting.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/CAFBEvLtEbUhutdYcBc58Zme7tCOT-nVacod2tAWPMuYq954w_Q%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/prometheus-users/CAFBEvLtEbUhutdYcBc58Zme7tCOT-nVacod2tAWPMuYq954w_Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAFBEvLseLU5B%3Dv%2B2KxQ7%3D_2KdyhxhrcPSU-w2PjL9OYtJg6seQ%40mail.gmail.com.
