Hi, Indeed that's not ideal and annotations and such will only be available in the source Prometheus servers (and then pushed to the Alertmanagers). I'm not sure what exactly your network constraints are, but they seem to prohibit Alertmanager in each location talking directly to Slack. Would they also prohibit each location's Alertmanager pushing additionally to a global Alertmanager? Because if that's possible, you could point each location's Prometheus to both a local and a global Alertmanager, and both would receive all alerting info, and would be able to route alerts differently.
Julius On Fri, Jan 15, 2021 at 9:39 AM 'Dennis Murtic' via Prometheus Users < [email protected]> wrote: > Hi all, > > Currently we have different environments where Prometheus and Alertmanager > are used. These environments are also protected by certain network > settings, so we need to use PushProx. To collect data from all > environments, we use a "global" Prometheus server that federates with the > environments Prometheus server. > Alarms are emailed directly from the environments. This means that the > alarm rules are also configured per environment. Now we would like to send > the alerts from all environments to one of our Slack channels as well. > Since the environments are protected by certain network settings, it is not > possible to send the alarms from the environments directly to our Slack > channel. One of our ideas is to configure an alert in the "global" alert > manager that looks like this: > > - alert: AlertName > expr: > ALERTS{alertname!="AlertName",alertstate="firing",severity="critical"} > labels: > severity: critical > annotations: > alertname: '{{ $labels.alertname }}' > summary: Alert {{ $labels.alertname }} fired. > > This seems to work, but unfortunately this is not the best solution as > alarms often switch between firing/resolved state and also the summary is > not available here. > > Does anyone have any ideas on how we can improve this? > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAFERo3qk1zjB3krqNePeomzgvBXdVzTyTVd%3DkD9SbCJuM3WWhA%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CAFERo3qk1zjB3krqNePeomzgvBXdVzTyTVd%3DkD9SbCJuM3WWhA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5yccSbxFP6z0KQnjyPf7UEJqfS4YiRs3pqBejRwFh7aRg%40mail.gmail.com.
