Re: [prometheus-users] AlertManager firing duplicate alerts

[Sagar]

Thank you so much for the pointer Christian.
It resolved the issue.

Thanks
Sunil Sagar

On Sun, Apr 5, 2020 at 12:58 AM Christian Hoffmann <
m...@hoffmann-christian.info> wrote:

> On 4/4/20 6:42 PM, Sagar wrote:
> >  relabel_configs:
> >  - action: drop
> >source_labels: [replica]
> >regex: (.*)
> [...]
> >
> > For other server, replica is secondary
> > I want to drop only label in alert manager , but it drops entire alert
> > in alert_manager.
>
> Ah, now I see the problem. You are using action: drop -- this means
> exactly that: dropping the whole alert.
> You want to use action: labeldrop instead.
>
> Kind regards,
> Christian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CANegE52RWJ2mCtO7NSQpYu5dHzhGY9DzVmczYHrsjRYVhFcy9g%40mail.gmail.com.

Re: [prometheus-users] AlertManager firing duplicate alerts

[Christian Hoffmann]

On 4/4/20 6:42 PM, Sagar wrote:
>      relabel_configs:
>      - action: drop
>        source_labels: [replica]
>        regex: (.*)
[...]
> 
> For other server, replica is secondary 
> I want to drop only label in alert manager , but it drops entire alert
> in alert_manager. 

Ah, now I see the problem. You are using action: drop -- this means
exactly that: dropping the whole alert.
You want to use action: labeldrop instead.

Kind regards,
Christian

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e180cb19-ba76-6b3a-1098-60f33a2d2cca%40hoffmann-christian.info.

Re: [prometheus-users] AlertManager firing duplicate alerts

[Sagar]

Hi Christian,

Thank you for the helping hand .

My prometheus.yml file looks like this  :

global:
  external_labels:
 dc: europe
 replica: primary

alerting:
  alertmanagers:
- static_configs:
  - targets:
- am:9093
 relabel_configs:
 - action: drop
   source_labels: [replica]
   regex: (.*)

rule_configs:
  - prom_rules.yml

scrap_configs:

  - job_name: 'prometheus'

.
.
.

For other server, replica is secondary
I want to drop only label in alert manager , but it drops entire alert in
alert_manager.

In Alert-Manager , I see details in boxes like as given

dc=europeinstance=servername  replica=primary
dc=europeinstance=servername  replica=secondary

I want the output like below, to avoid duplicate entry from secondary server
dc=europeinstance=servername

Please suggest

Thanks

On Sat, Apr 4, 2020 at 3:04 AM Christian Hoffmann <
m...@hoffmann-christian.info> wrote:

> Hi Sunil,
>
> On 4/3/20 6:00 PM, Sagar wrote:
> > For more information , there are more global labels (such as
> > environment, cluster , etc) apart from replica .
> >
> > Do I need to remove other global variables and maintain only one label
> > as replica .
> You need to drop those labels which are used to distinguish different
> Prometheus instances which scrape the same targets.
>
> Some example:
> If you've got two Prometheus servers scraping all your database servers
> and are adding external labels such as cluster=database and replica=A
> for the one and replica=B for the other and if you've got another pair
> of Prometheus servers for your web servers with external labels
> cluster=web and replica=A and replica=B again, then you would only drop
> the replica label, not the cluster label.
>
> > I also tried with options given below ( from the link
> > : https://github.com/prometheus/prometheus/issues/3239 )
> >
> > |alerting: alert_relabel_configs:- action: drop source_labels:
> > [replica] regex: '1'|
> >
> > Please advise , thanks
> I don't think you should have regex: 1 here? Just drop the regex line so
> that it will default to '(.*)' which matches all values.
>
> If this doesn't help, can you share your actual config + the behaviour
> you are seeing?
>
> > Is it possible to query alertmanager alerts like metrics in prometheus ?
> > Like http:/server:9093/mertrics or something else .
> Alertmanager does have metrics, but it does not export alerts there.
> However, there is an API for reading them programatically.
>
>
> https://petstore.swagger.io/?url=https://raw.githubusercontent.com/prometheus/alertmanager/master/api/v2/openapi.yaml#/alert
>
>
> Kind regards,
> Christian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CANegE51_xR_HkC4GOG%2BNpJTWSJSy%2BWZ%3DpfO8DFXsKt2Ob_%2Br6w%40mail.gmail.com.

Re: [prometheus-users] AlertManager firing duplicate alerts

[Christian Hoffmann]

Hi Sunil,

On 4/3/20 6:00 PM, Sagar wrote:
> For more information , there are more global labels (such as
> environment, cluster , etc) apart from replica . 
> 
> Do I need to remove other global variables and maintain only one label
> as replica . 
You need to drop those labels which are used to distinguish different
Prometheus instances which scrape the same targets.

Some example:
If you've got two Prometheus servers scraping all your database servers
and are adding external labels such as cluster=database and replica=A
for the one and replica=B for the other and if you've got another pair
of Prometheus servers for your web servers with external labels
cluster=web and replica=A and replica=B again, then you would only drop
the replica label, not the cluster label.

> I also tried with options given below ( from the link
> : https://github.com/prometheus/prometheus/issues/3239 )
> 
> |alerting: alert_relabel_configs:- action: drop source_labels:
> [replica] regex: '1'|
> 
> Please advise , thanks 
I don't think you should have regex: 1 here? Just drop the regex line so
that it will default to '(.*)' which matches all values.

If this doesn't help, can you share your actual config + the behaviour
you are seeing?

> Is it possible to query alertmanager alerts like metrics in prometheus ?
> Like http:/server:9093/mertrics or something else . 
Alertmanager does have metrics, but it does not export alerts there.
However, there is an API for reading them programatically.

https://petstore.swagger.io/?url=https://raw.githubusercontent.com/prometheus/alertmanager/master/api/v2/openapi.yaml#/alert


Kind regards,
Christian

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/72ba1626-f957-28ff-35db-9b89e72b75a1%40hoffmann-christian.info.

Re: [prometheus-users] AlertManager firing duplicate alerts

[Sagar]

Hi Christian ,

Thank you for helping me on this. For some reason, I am not able to drop
these global labels .
I tried config as mentioned in link you shared .

For more information , there are more global labels (such as environment,
cluster , etc) apart from replica .

Do I need to remove other global variables and maintain only one label as
replica .

I also tried with options given below ( from the link :
https://github.com/prometheus/prometheus/issues/3239 )

alerting:
  alert_relabel_configs:
- action: drop
  source_labels: [replica]
  regex: '1'

Please advise , thanks

Is it possible to query alertmanager alerts like metrics in prometheus ?
Like http:/server:9093/mertrics or something else .

Thanks,
Sunil Sagar

On Thu, Mar 26, 2020 at 5:37 AM Christian Hoffmann <
m...@hoffmann-christian.info> wrote:

> Hi,
>
> you seem to be using external_labels without alert_relabel_configs to
> drop this label from your alerts again. Therefore, your alerts will have
> different labels and will not be de-duplicated.
>
> See this blog post:
>
> https://www.robustperception.io/high-availability-prometheus-alerting-and-notification
>
> It has an example for the dc label (where you would need replica).
>
> Kind regards,
> Christian
>
> On 3/25/20 5:17 AM, sunil sagar wrote:
> > Hi ,
> >
> > I have Prometheus environment in HA mode . And AlertManager is also in
> > HA mode .
> > I am receiving duplicate alerts .
> > When I start both the prometheus , because of global label with
> > different replica name , getting duplicate alert . Please advise.
> >
> > Prometheus Config:
> >
> > Prometheus Node1:
> > global:
> >external_labels:
> >   replica: 1
> >
> > alerting:
> >   alertmanagers:
> >  -static_configs:
> >  - targets:
> >   - alertmanager1:9093
> >   - alertmanager2:9093
> >
> > --
> > Prometheus Node2:
> > global:
> >external_labels:
> >   replica: 1
> >
> > alerting:
> >   alertmanagers:
> >  -static_configs:
> >  - targets:
> >   - alertmanager1:9093
> >   - alertmanager2:9093
> > -
> > Sample AlertManager rule:
> > expr: max(up == 0 ) by (host)
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to prometheus-users+unsubscr...@googlegroups.com
> > .
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/prometheus-users/c1b788da-59c0-439c-8fc2-d1ded5f0bf46%40googlegroups.com
> > <
> https://groups.google.com/d/msgid/prometheus-users/c1b788da-59c0-439c-8fc2-d1ded5f0bf46%40googlegroups.com?utm_medium=email_source=footer
> >.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CANegE50uJc2rCk-9axLF1WATWorw8%2B_izb12o3Wqet5i3x9UjA%40mail.gmail.com.

Re: [prometheus-users] AlertManager firing duplicate alerts

[Christian Hoffmann]

Hi,

you seem to be using external_labels without alert_relabel_configs to
drop this label from your alerts again. Therefore, your alerts will have
different labels and will not be de-duplicated.

See this blog post:
https://www.robustperception.io/high-availability-prometheus-alerting-and-notification

It has an example for the dc label (where you would need replica).

Kind regards,
Christian

On 3/25/20 5:17 AM, sunil sagar wrote:
> Hi , 
> 
> I have Prometheus environment in HA mode . And AlertManager is also in
> HA mode . 
> I am receiving duplicate alerts . 
> When I start both the prometheus , because of global label with
> different replica name , getting duplicate alert . Please advise. 
> 
> Prometheus Config:
> 
> Prometheus Node1:
> global:
>    external_labels:
>       replica: 1
> 
> alerting:
>   alertmanagers:
>      -static_configs:
>          - targets: 
>               - alertmanager1:9093
>               - alertmanager2:9093
> 
> --
> Prometheus Node2:
> global:
>    external_labels:
>       replica: 1
> 
> alerting:
>   alertmanagers:
>      -static_configs:
>          - targets: 
>               - alertmanager1:9093
>               - alertmanager2:9093
> -
> Sample AlertManager rule:
> expr: max(up == 0 ) by (host)
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-users+unsubscr...@googlegroups.com
> .
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c1b788da-59c0-439c-8fc2-d1ded5f0bf46%40googlegroups.com
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d27b20d4-9732-298f-85e1-d94a02a66db7%40hoffmann-christian.info.

Re: [prometheus-users] AlertManager firing duplicate alerts

[sunil sagar]

Hi All , 

Corrected on error :
Node 2 is marked as replica2 
> 
> Prometheus Config:
> 
> Prometheus Node1:
> global:
>external_labels:
>   replica: 1
> 
> alerting:
>   alertmanagers:
>  -static_configs:
>  - targets: 
>   - alertmanager1:9093
>   - alertmanager2:9093
> 
> --
> Prometheus Node2:
> global:
>external_labels:
>   replica: 2
> 
> alerting:
>   alertmanagers:
>  -static_configs:
>  - targets: 
>   - alertmanager1:9093
>   - alertmanager2:9093
> -



Thanks

> On 25 Mar 2020, at 12:18 PM, sunil sagar  wrote:
> 
> 
> Hi , 
> 
> I have Prometheus environment in HA mode . And AlertManager is also in HA 
> mode . 
> I am receiving duplicate alerts . 
> When I start both the prometheus , because of global label with different 
> replica name , getting duplicate alert . Please advise. 
> 
> Prometheus Config:
> 
> Prometheus Node1:
> global:
>external_labels:
>   replica: 1
> 
> alerting:
>   alertmanagers:
>  -static_configs:
>  - targets: 
>   - alertmanager1:9093
>   - alertmanager2:9093
> 
> --
> Prometheus Node2:
> global:
>external_labels:
>   replica: 1
> 
> alerting:
>   alertmanagers:
>  -static_configs:
>  - targets: 
>   - alertmanager1:9093
>   - alertmanager2:9093
> -
> Sample AlertManager rule:
> expr: max(up == 0 ) by (host)
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/c1b788da-59c0-439c-8fc2-d1ded5f0bf46%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/512BCF6B-BDA3-436A-BD9C-D2C0F3522F92%40gmail.com.