thanks Ben! it looks like a good suggestion. still learning prom SQL, see
a function called absent. do this function help for my case?
在 2020年6月3日星期三 UTC+8下午4:05:24，Ben Kochie写道：
>
> You can use something like
> `avg_over_time(node_processes_state{state='D'}[10m])` to smooth over missed
> scrapes. Depending on how sensitive you want this to be, you can also do
> `max_over_time()`.
>
> On Wed, Jun 3, 2020 at 9:49 AM 林浩 <[email protected] <javascript:>> wrote:
>
>>
>> We use node export to monitor os D state process, when D state process
>> number > 500, it triggers pager alert rule like these
>>
>> - alert: Node_Process_In_D_State_Count_Critical
>> expr: node_processes_state{state='D'} > 500
>> for: 10m
>>
>> but the problem is when OS running into problem status (too much D state
>> process), looks like node export agent also running in problem status, it
>> can NOT report correct D state process metric to Prometheus server.
>> from the below screenshot, we can see some data points missing. This
>> causes alert flapping, when data missing, the alert gets resolved.
>>
>> is any way to avoid alert auto resolved when some data points missed?
>>
>> [image: Jietu20200603-154527.jpg]
>>
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/f8560b07-9b00-4dfc-9671-667368ddd530%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/f8560b07-9b00-4dfc-9671-667368ddd530%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/9c94720c-7347-448f-8790-26f745739e9e%40googlegroups.com.
