This is very strange, to say the least. With the >= 90 filter on the top level of the expression, the alerting rule should never return series that have a sample value below 90, and thus $value should never be below 90 either. In case this is really not a misconfiguration somewhere (please double-check, and also try the latest Prometheus version), it would be interesting if you can make this a reproducible case somehow for a bug report.
On Fri, Apr 24, 2020 at 11:18 AM J Houer <[email protected]> wrote: > We use a Prometheus alert (and node-exporter) to check whether we are > running out of memory on a node. > > > Issue: In many cases I get an alert with a $value that is below the > threshold value in the expression.The expression is: > > > alert: GettingOutOfMemory >> expr: max(sum >> by(instance) ((((node_memory_MemTotal_bytes) - (node_memory_MemFree_bytes >> + node_memory_Buffers_bytes >> + node_memory_Cached_bytes)) / (node_memory_MemTotal_bytes)) * 100)) >= 90 >> for: 5m >> labels: >> severity: warning >> annotations: >> description: Docker Swarm node {{ $labels.instance }} memory usage is at >> {{ humanize $value}}%. >> summary: Memory is getting low for Swarm node '{{ $labels.node_name }}' >> >> > I get messages saying that we ran out of memory at e.g. 63%. So that is > the value of the $value. This is clearly below the 90% threshold. > > Why do I get this alert even though the $value is below the threshold? > > > How can I repair this Prometheus alert rule so I will only get only alerts > when the $value is above the threshold? > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/6a2cc11f-b733-4a66-87b5-55e8355d7ebb%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/6a2cc11f-b733-4a66-87b5-55e8355d7ebb%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BT6YoxMZGx%2Bmtp_4AgrjNiwnF3tqaTR3yZ0ht0C4LRtVAsr2Q%40mail.gmail.com.
