Events are not metrics. For the behaviour you want, you should be looking at an event-based system like Loki.
On Thursday, 28 July 2022 at 12:14:39 UTC+1 [email protected] wrote: > Hello Stuart > I'm sorry I couldn't ask my question properly > Actually, I use prometheus/alertmanager as an event pipeline to alert > every event that occurs. > > For example, I show two of the rules that are exported from the log > exporter service: > - name: plate > rules: > - alert: "plate" > expr: 'plate_log {plate_number="123456877Lay"}' > for: 1s > annotations: > title: "plate detection {{ $labels.model_camera_id }}" > description: "plate detection with confidence : {{ > $labels.confidence }} " > labels: > severity: "critical" > type: "plate" > > - name: human > rules: > - alert: "human" > expr: 'number_of_Human > 15' > for: 1s > annotations: > title: "human detection {{ $labels.model_camera_id }}" > description: "human detection with confidence : {{ > $labels.confidence }} " > labels: > severity: "critical" > type: "human" > > > Also, the alertmanager configuration is as follows: > > global: > > route: > receiver: webhook > group_by: ["alertname"] > group_wait: 1s > group_interval: 1s > # repeat_interval: 6d > routes: > - receiver: webhook > continue: true > Receivers: > - name: webhook > webhook_configs: > - send_resolved: false > http_config: {} > url: "http://192.168.10.20:7000/visual"; > max_alerts: 0 > - url: "http://192.168.10.20:9200/alerts/_doc"; > send_resolved: false > > My problem is exactly that if 5 alerts are hooked at different times, for > the sixth log, all the previous 5 logs are also hooked. > I felt that because the logs are still in firing mode after the hook, they > are sent again to Front and Elastic with new logs. > > Stuart Clark در تاریخ یکشنبه ۲۴ ژوئیهٔ ۲۰۲۲ ساعت ۱۹:۵۹:۳۵ (UTC+4:30) نوشت: > >> On 24/07/2022 11:10, Milad Devops wrote: >> > hi all >> > I use Prometheus to create alert rules and hook alerts using >> alertmanager. >> > My scenario is as follows: >> > - The log publishing service sends logs to Prometheus Exporter >> > - Prometheus takes the logs every second and matches them with our >> rules >> > - If the log applies to our rules, the alertmanager sends an alert to >> > the frontend application. It also saves the alert in the elastic >> > >> > My problem is that when sending each alert, all the previous alerts >> > are also stored in Elastic in the form of a single log and sent to my >> > front service as a notification (web hook). >> > >> > Is there a way I can change the alert status to resolved after the >> > hook so that it won't be sent again on subsequent hooks? >> > Or delete the previous logs completely after the hook from Prometheus >> > Or any other suggested way you have >> > Thank you in advance >> >> I'm not sure I really understand what you are asking due to your >> mentioning of logs. >> >> Are you saying that you are using an exporter (for example mtail) which >> is consuming logs and then generating metrics? >> >> When you create an alerting rule in Prometheus it performs the PromQL >> query given, and if there are any results an alert is fired. Once the >> PromQL query stops returning results (or has a different set of time >> series being returned) the alert is resolved. >> >> So for example if you had a simple query that said "alert if the number >> of error logs [stored in a counter metric] increases by 5 or more in the >> last 5 minutes" as soon as the metric returned an increase of at least 5 >> over the last 5 minutes it would fire. It would then continue to fire >> until that is no longer true - so if the counter kept recording error >> log lines such that the increase was still over 5 per 5 minutes it would >> keep firing. It would only resolve once there were no more than 5 new >> long lines recorded over the past 5 minutes. >> >> Alertmanager just routes alerts that are generated within Prometheus to >> other notification/processing systems, such as email or webhooks. It >> would normally fire the webhook once the alert starts firing, and then >> periodically (if it keeps firing, at a configurable interval) and then >> finally (optionally) once it resolves. This is a one-way process - >> nothing about the notification has any impact on the alert firing or >> not. Only the PromQL query controls the alert. >> >> I'm not sure if that helps. >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/594cd4ae-49f9-495a-9856-f7e86b75f64dn%40googlegroups.com.
