Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-11-29 Thread Won
Hi, I checked that both of the methods you propose work well. After I add 'should_delete_outdated_entities' function to InstanceDriver, it took about 10 minutes to clear the old Instance. And I added two sentences you said to Nova-cpu.conf, so the vitrage collector get notifications well. Thank y

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-11-22 Thread Ifat Afek
Hi, A deleted instance should be removed from Vitrage in one of two ways: 1. By reacting to a notification from Nova 2. If no notification is received, then after a while the instance vertex in Vitrage is considered "outdated" and is deleted Regarding #1, it is clear from your logs that you don't

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-11-15 Thread Ifat Afek
On Thu, Nov 15, 2018 at 10:28 AM Won wrote: > Looking at the logs, I see two issues: >> 1. On ubuntu server, you get a notification about the vm deletion, while >> on compute1 you don't get it. >> Please make sure that Nova sends notifications to 'vitrage_notifications' >> - it should be configur

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-11-08 Thread Ifat Afek
Hi, We solved the timestamp bug. There are two patches for master [1] and stable/rocky [2]. I'll check the other issues next week. Regards, Ifat [1] https://review.openstack.org/#/c/616468/ [2] https://review.openstack.org/#/c/616469/ On Wed, Oct 31, 2018 at 10:59 AM Won wrote: > [image

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-11-01 Thread Ifat Afek
Hi, On Wed, Oct 31, 2018 at 11:00 AM Won wrote: > Hi, > >> >> This is strange. I would expect your original definition to work as well, >> since the alarm key in Vitrage is defined by a combination of the alert >> name and the instance. We will check it again. >> BTW, we solved a different bug

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-10-31 Thread Won
Hi, > > This is strange. I would expect your original definition to work as well, > since the alarm key in Vitrage is defined by a combination of the alert > name and the instance. We will check it again. > BTW, we solved a different bug related to Prometheus alarms not being > cleared [1]. Could

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-10-29 Thread Ifat Afek
Hi, On Fri, Oct 26, 2018 at 10:34 AM Won wrote: > > I solved the problem of not updating the Prometheus alarm. > Alarms with the same Prometheus alarm name are recognized as the same > alarm in vitrage. > > --- alert.rules.yml > groups: > - name: alert.rules > rules: > - alert: InstanceD

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-10-10 Thread Ifat Afek
Hi Won, On Wed, Oct 10, 2018 at 11:58 AM Won wrote: > > my prometheus version : 2.3.2 and alertmanager version : 0.15.2 and I > attached files.(vitrage collector,graph logs and apache log and > prometheus.yml alertmanager.yml alarm rule file etc..) > I think the problem that resolved alarm does

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-10-04 Thread Ifat Afek
Hi, Can you please give us some more details about your scenario with Prometheus? Please try and give as many details as possible, so we can try to reproduce the bug. What do you mean by “if the alarm is resolved, the alarm manager makes a silence, or removes the alarm rule from Prometheus”? the

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-10-03 Thread Won
Thank you for your reply Ifat. The alertmanager.yml file already contains 'send_resolved:true'. However, the alarm does not disappear from the alarm list and the entity graph even if the alarm is resolved, the alarm manager makes a silence, or removes the alarm rule from Prometheus. The only way t

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-10-03 Thread Ifat Afek
Hi, In the alertmanager.yml file you should have a receiver for Vitrage. Please verify that it includes "send_resolved: true". This is required for Prometheus to notify Vitrage when an alarm is resolved. The full Vitrage receiver definition should be: - name: ** webhook_configs: - url: **

[openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

2018-10-01 Thread Won
I have some problems with Prometheus alarms in vitrage. I receive a list of alarms from the Prometheus alarm manager well, but the alarm does not disappear when the problem(alarm) is resolved. The alarm that came once in both the alarm list and the entity graph does not disappear in vitrage. The a