These posts may be helpful:
https://www.robustperception.io/absent-alerting-for-scraped-metrics
https://www.robustperception.io/absent-alerting-for-jobs
https://www.robustperception.io/existential-issues-with-metrics
The expression
absent(mymetric{foo="bar",baz="qux"})
"returns an empty vector if the vector passed to it has any elements and a
1-element vector with the value 1 if the vector passed to it has no
elements" [ref
<https://prometheus.io/docs/prometheus/latest/querying/functions/#absent>]
So you should be able to build something like this:
(mymetric{foo="bar",baz="qux"} * 0) or
absent(mymetric{foo="bar",baz="qux"})
That "or" expression [ref
<https://prometheus.io/docs/prometheus/latest/querying/operators/#logical-set-binary-operators>]
should evaluate to either
{foo="bar",baz="qux"} 0
{foo="bar",baz="qux"} 1
But it really depends on the situation. In some cases, the metric "up" is
all you need. It's 1 when a scrape was successful, and 0 when the scrape
failed. Even if the scrapes are successful, "up" is a useful metric to
combine with absent(), as shown in the blog posts linked above.
On Wednesday, 26 January 2022 at 11:13:45 UTC [email protected]
wrote:
> Thanks for the great explanation Brian.
> absent() looks like the right solution but the documentation about empty
> vectors confuses me slightly. What I would ideally like is to graph the
> absent() as two distinct values - is there a way to return that as a query
> or do I have to address this in my graphing tool (Grafana)?
>
> On Tuesday, January 25, 2022 at 7:35:21 PM UTC Brian Candler wrote:
>
>> Prometheus does not store or return "null".
>>
>> Prometheus will store a "staleness marker" when it performs a scrape and
>> doesn't find a timeseries (i.e. metric+label set) which was present in the
>> previous scrape. Internally, it's stored as a special kind of
>> floating-point NaN. However it's not returned in queries. Instead, the
>> timeseries just "disappears" as if it didn't exist.
>>
>> A timeseries consists of a series of values V1, V2, V3 .. at timestamps
>> T1, T2, T3 ... So when you perform an instant query for some arbitrary
>> time T, then prometheus has to look back in time to find the nearest data
>> point at or before time T. It will look back up to --query.lookback-delta
>> (by default 5m), but if it finds a staleness marker, it just doesn't return
>> any value, i.e. that timeseries is excluded from the vector result. This
>> is all described here
>> <https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness>
>> .
>>
>> If you want to look gaps in a particular timeseries, you can build a
>> query using absent()
>> <https://prometheus.io/docs/prometheus/latest/querying/functions/#absent>,
>> giving a specific timeseries (metric+labels) that you're looking for, and
>> prometheus will tell you when the time series doesn't exist (i.e. there is
>> no data point within the previous 5 minutes *or* the series has been marked
>> stale)
>> absent{foo{instance="bar",job="baz"})
>>
>> If you want to see when arbitrary timeseries disappear, without
>> hardcoding specific label sets, you can try something like this:
>>
>> foo offset 5m unless foo # it existed 5 minutes ago, but doesn't
>> exist now
>>
>> present_over_time(foo[24h] offset 5m) unless foo # existed any
>> time in the last 24 hours until 5 minutes ago, but doesn't exist now
>>
>> On Tuesday, 25 January 2022 at 18:59:23 UTC [email protected]
>> wrote:
>>
>>> I know in Grafana I can choose to display NULL as zero or as missing but
>>> I actually want to graph when a value is null - to visualise downtime where
>>> a series has gaps. Can I craft a query which will spit out two values for
>>> NULL / not NULL?
>>>
>>> Thanks.
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/be31694c-f2ec-42c8-a35d-a9f5c0eb8cf3n%40googlegroups.com.
