For this scenario, I think we should be fine. The metadata we're applying is stable (it shouldn't change over the lifetime of the alerts unless we need to reconfigure the entire Prometheus instance) and it applying to every alert is actually desired in this case, because we can't access the alerts any other way. If we could apply labels/annotations more granularly, we would do so, but since we can't directly modify the alerts themselves, the external labels config is our best option currently for doing any sort of enrichment of the metadata on the labels, so we'll take what we can get!
On Sun, Jan 17, 2021 at 4:02 AM Julius Volz <[email protected]> wrote: > Hi Brian, > > An alert is identified by its unique set of labels, so the labels really > serve as an identification, and changing / adding / removing any label will > semantically mean that it's a different alert than before (with all the > effects that has on routing and grouping in Alertmanager). Annotations are > non-identifying pieces of extra information on alerts that do not affect > routing or grouping at all, so those are indeed likely what you want. What > you want to do might work with just labels if your metadata has a stable > relationship to existing alerts that wouldn't cause frequent changes to > their label set identities (e.g. if it's just a few extra attributes that > rarely change). What is the metadata you want to attach? > > External labels are applied to every outgoing alert of a Prometheus server > equally (not just for some alerts), is that what you want? > > Regards, > Julius > > On Fri, Jan 15, 2021 at 4:38 PM Brian Gottfried <[email protected]> > wrote: > >> We're forwarding our alerts out to an external system that requires some >> additional metadata and I'm trying to understand the effect of providing >> this metadata as annotations vs labels on each alert. >> >> Context: We're utilizing both custom alerts and the default alerts that >> come with Openshift. The wrinkle is that those Openshift default alerts are >> Operator-managed, so we can't modify them directly to add the metadata as >> annotations (which from what I've found is the correct way to add metadata >> that you want to be the same across all instances of an alert). However, we >> can override the Openshift Prometheus' settings to add the values via the >> external labels field. Will this cause any issues with alerts being grouped >> together oddly or not being sent? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/14d57178-e2ff-4902-a270-655f4cb5e04fn%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/14d57178-e2ff-4902-a270-655f4cb5e04fn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGHJLg2rr8W43WsyPNmUXpxTt0ND%3DYMVkvKd_Q%2Bp1_NPHtkxCQ%40mail.gmail.com.
