Thanks for the quick response Stuart. One of our specific use cases (although there will be more over time) would be something where a first or second level support team escalates an issue they can't solve t to the engineers responsible for the product. In this case, there would be no metric as this is an event that could happen at any time for which we don't really want a metric. Triggering the alert via alertmanager seemed a logical choice as it already handles the logic of the routing to the necessary destinations (email, webhook, victorops, etc). All the user would have to do us run the amtool command, with the necessary labels and wouldn't have to worry about any other specifics.
Based on your explanation, I now understand alertmanager can't really be used that way. Could you show me where in the AM sourcecode that it will close an alert unless it is continuously notified by Prometheus? I'd like to know for my own personal knowledge. Now having considered these facts, do you have any suggestions based on this example? Is this just something we should handle separately with another custom application? If that's the case, it's a bit discouraging as now that means we have to handle the logic of alert routing in more than one location. Al On Monday, December 14, 2020 at 12:52:53 PM UTC-5 Stuart Clark wrote: > On 2020-12-14 17:05, Al wrote: > > Hi > > > > I realize alert conditions in a Promertheus ecosystem should be > > triggered from a prometheus instance itself although there is the > > "amtool alert add" command that can be used to manually trigger an > > alert. Is this something which is commonly used in production > > use-cases? I can see a benefit to using this command as I could still > > allow users to trigger alerts in a standardized way, but without > > having to have specific pre-defined alerting conditions. There may > > also be situations where there is no metric collected but only an > > alert to be triggered in the situation a specific event occurs. > > > > From my understanding, when prometheus fires an alert, it will send > > the payload to all instances of alert manager with in the cluster and > > then they will handle which instance will actually route the alert to > > the final destination (e.g.: Victorops, email, webook, etc). If this > > is in fact correct, does this mean that amtool should also send the > > alert to all alertmanager instances within the cluster? > > > > I appreciate any clarification you can provide me with. > > > > That command is only intended for testing. Alerts aren't a one-off API > call from Prometheus to Alertmanager. Instead Prometheus will repeatedly > call every single Alertmanager periodically until the alert is cleared. > If Alertmanager stops receiving these updates it will mark the alert as > resolved. > > Alerts in the Prometheus world are triggered based on the evaluation of > alerting rules, which themselves are queries which interrogate metrics. > Therefore every alert would be based on some sort of source metric > (there are a few exceptions, such as having an alert which always fires > to check the alerting pipeline for example). > > For one of the example use cases you gave you said an alert should be > triggered if an event happens. Prometheus itself isn't an event system, > but you can create metrics from events. So in that case you'd have a > metric that could be a counter of the number of events that have > happened. Then your alert would fire when that value increases (for > example). > > Are you able to give some more information on what use cases you are > trying to handle? > > -- > Stuart Clark > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/74029e3f-d642-4174-9a19-646c23618430n%40googlegroups.com.
