[openstack-dev] FW: Change in openstack/masakari-monitors[master]: Introspective Instance Monitoring through QEMU Guest Agent

[Kwan, Louie]

Thanks again Tushar and Adam for reviewing 534958.



Anything else to get Workflow to +1?



Thanks.

Louie



-Original Message-
From: Tushar Patil (Code Review) [mailto:rev...@openstack.org]
Sent: Tuesday, July 10, 2018 10:21 PM
To: Kwan, Louie
Cc: Friesen, Chris; Tim Bell; Waines, Greg; Li Yingjun; Sampath Priyankara 
(samP); wangqiang-bj; Jean-Philippe Evrard; Young, Ken; Tushar Patil; Andrew 
Beekhof; Abhishek Kekane; zhangyangyang; takahara.kengo; Rikimaru Honjo; Dinesh 
Bhor; Michele Baldessari; Adam Spiers
Subject: Change in openstack/masakari-monitors[master]: Introspective Instance 
Monitoring through QEMU Guest Agent



Tushar Patil has posted comments on this change. ( 
https://review.openstack.org/534958 )



Change subject: Introspective Instance Monitoring through QEMU Guest Agent

..





Patch Set 9: Code-Review+2



LGTM. Thank you.



--

To view, visit https://review.openstack.org/534958

To unsubscribe, visit https://review.openstack.org/settings



Gerrit-MessageType: comment

Gerrit-Change-Id: I9efc6afc8d476003d3aa7fee8c31bcaa65438674

Gerrit-PatchSet: 9

Gerrit-Project: openstack/masakari-monitors

Gerrit-Branch: master

Gerrit-Owner: Louie Kwan 

Gerrit-Reviewer: Abhishek Kekane 

Gerrit-Reviewer: Adam Spiers 

Gerrit-Reviewer: Andrew Beekhof 

Gerrit-Reviewer: Chris Friesen 

Gerrit-Reviewer: Dinesh Bhor 

Gerrit-Reviewer: Greg Waines 

Gerrit-Reviewer: Hieu LE 

Gerrit-Reviewer: Jean-Philippe Evrard 

Gerrit-Reviewer: Ken Young 

Gerrit-Reviewer: Li Yingjun 

Gerrit-Reviewer: Louie Kwan 

Gerrit-Reviewer: Michele Baldessari 

Gerrit-Reviewer: Rikimaru Honjo 

Gerrit-Reviewer: Sampath Priyankara (samP) 

Gerrit-Reviewer: Tim Bell 

Gerrit-Reviewer: Tushar Patil 

Gerrit-Reviewer: Tushar Patil 

Gerrit-Reviewer: Zuul

Gerrit-Reviewer: takahara.kengo 

Gerrit-Reviewer: wangqiang-bj 

Gerrit-Reviewer: zhangyangyang 

Gerrit-HasComments: No
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [masakari] Introspective Instance Monitoring through QEMU Guest Agent

[Kwan, Louie]

Thanks Tushar Patil for the +1 for 547118.

In regards of the following review:
https://review.openstack.org/#/c/534958/

Any more comment?

Thanks.
Louie

From: Tushar Patil (Code Review) [rev...@openstack.org]
Sent: Tuesday, July 03, 2018 8:48 PM
To: Kwan, Louie
Cc: Tim Bell; zhangyanying; Waines, Greg; Li Yingjun; wangqiang-bj; Tushar 
Patil; Ken Young; NTT system-fault-ci masakari-integration-ci; wangqiang; 
Abhishek Kekane; takahara.kengo; Rikimaru Honjo; Adam Spiers; Sampath 
Priyankara (samP); Dinesh Bhor
Subject: Change in openstack/masakari[master]: Introspective Instance 
Monitoring through QEMU Guest Agent

Tushar Patil has posted comments on this change. ( 
https://review.openstack.org/547118 )

Change subject: Introspective Instance Monitoring through QEMU Guest Agent
..


Patch Set 3: Workflow+1

--
To view, visit https://review.openstack.org/547118
To unsubscribe, visit https://review.openstack.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I9efc6afc8d476003d3aa7fee8c31bcaa65438674
Gerrit-PatchSet: 3
Gerrit-Project: openstack/masakari
Gerrit-Branch: master
Gerrit-Owner: Louie Kwan 
Gerrit-Reviewer: Abhishek Kekane 
Gerrit-Reviewer: Adam Spiers 
Gerrit-Reviewer: Dinesh Bhor 
Gerrit-Reviewer: Greg Waines 
Gerrit-Reviewer: Hieu LE 
Gerrit-Reviewer: Ken Young 
Gerrit-Reviewer: Li Yingjun 
Gerrit-Reviewer: Louie Kwan 
Gerrit-Reviewer: NTT system-fault-ci masakari-integration-ci 

Gerrit-Reviewer: Rikimaru Honjo 
Gerrit-Reviewer: Sampath Priyankara (samP) 
Gerrit-Reviewer: Tim Bell 
Gerrit-Reviewer: Tushar Patil 
Gerrit-Reviewer: Tushar Patil 
Gerrit-Reviewer: Zuul
Gerrit-Reviewer: takahara.kengo 
Gerrit-Reviewer: wangqiang 
Gerrit-Reviewer: wangqiang-bj 
Gerrit-Reviewer: zhangyanying 
Gerrit-HasComments: No

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Telemetry] [ceilometer] Ceilometer-file-publisher-compression-csv-format

[Kwan, Louie]

Hi All,

Any weekly meeting for Telemetry?  I would like to discuss what we can do for 
the next step for the following review?

https://review.openstack.org/#/c/562768/

Ping in the IRC a few times and please advice the next step.

Thanks.
Louie


From: Kwan, Louie
Sent: Friday, May 04, 2018 10:03 AM
To: openstack-dev@lists.openstack.org; julien.dan...@enovance.com
Subject: [openstack-dev] [Telemetry] [ceilometer] 
Ceilometer-file-publisher-compression-csv-format

Reaching out to Rocky PTL and others.

What could be the next step?

Thanks.
Louie

-Original Message-
From: Kwan, Louie [mailto:louie.k...@windriver.com]
Sent: Monday, April 23, 2018 4:10 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [ceilometer] 
Ceilometer-file-publisher-compression-csv-format

Submitted the following review on April 19,

https://review.openstack.org/#/c/562768/

Would like to know who else could be on the reviewer list and anything else for 
the next step?

Thanks.
Louie
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [masakari] Masakari Project Meeting time

[Kwan, Louie]

It seems most of the team members are on vacation this week.

By the way,  03:00 UTC is fine as well.

-Original Message-
From: Bhor, Dinesh [mailto:dinesh.b...@nttdata.com] 
Sent: Wednesday, April 25, 2018 9:09 PM
To: Kwan, Louie
Cc: Sampath Priyankara (samP); openstack-dev@lists.openstack.org; Young, Ken
Subject: Re: [openstack-dev] [masakari] Masakari Project Meeting time 

+1
This time may not fit for attendees who work in IST time zone as it will 07.30 
AM in the morning.

Thanks,
Dinesh

> On Apr 26, 2018, at 12:06 AM, Kwan, Louie <louie.k...@windriver.com> wrote:
>
> Sampath, Dinesh and others,
>
> It was a good meeting last week.
>
> As briefly discussed with Sampath, I would like to check whether we can 
> adjust the meeting time.
>
> We are at EST time zone, the meeting is right on our midnight time, 12:00 am.
>
> It will be nice if the meeting can be started ~2 hours earlier e.g. Could it 
> be  started at 02:00: UTC instead?
>
> Thanks.
> Louie
>
>

Disclaimer: This email and any attachments are sent in strictest confidence for 
the sole use of the addressee and may contain legally privileged,confidential, 
and proprietary data. If you are not the intended recipient,please advise the 
sender by replying promptly to this email and then delete and destroy this 
email and any attachments without any further use, copying or forwarding.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [masakari] Masakari Project Meeting time

[Kwan, Louie]

Sampath, Dinesh and others,

It was a good meeting last week.

As briefly discussed with Sampath, I would like to check whether we can adjust 
the meeting time. 

We are at EST time zone, the meeting is right on our midnight time, 12:00 am.

It will be nice if the meeting can be started ~2 hours earlier e.g. Could it be 
 started at 02:00: UTC instead?

Thanks.
Louie



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [masakari] Introspective Instance Monitoring through QEMU Guest Agent

[Kwan, Louie]

Submitted the following review on January 17, 2018,

https://review.openstack.org/#/c/534958/

Would like to know who else could be on the reviewer list ? or anything else is 
needed for the next step?

Also, I am planning to attend our coming Masakari Weekly meeting,  April 24, 
0400 UTC in #openstack-meeting  
and would like add an agenda item to  follow up how to move the  review forward.

Thanks.
Louie

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer] Ceilometer-file-publisher-compression-csv-format

[Kwan, Louie]

Submitted the following review on April 19, 

https://review.openstack.org/#/c/562768/

Would like to know who else could be on the reviewer list and anything else for 
the next step?

Thanks.
Louie
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [devstack] stable/queens: How to configure devstack to use openstacksdk===0.11.3 and os-service-types===1.1.0

[Kwan, Louie]

In the stable/queens branch, since openstacksdk0.11.3 and os-service-types1.1.0 
are described in openstack's upper-constraints.txt, 

https://github.com/openstack/requirements/blob/stable/queens/upper-constraints.txt#L411
https://github.com/openstack/requirements/blob/stable/queens/upper-constraints.txt#L297

If I do 

> git clone https://git.openstack.org/openstack-dev/devstack -b stable/queens

And then stack.sh

We will see it is using openstacksdk-0.12.0 and os_service_types-1.2.0

Having said that, we need the older version, how to configure devstack to use 
openstacksdk===0.11.3 and os-service-types===1.1.0

Thanks.
Louie


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-masakariclient] Installation issues

[Kwan, Louie]

It may be related to the profile changes?

> /usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/connection.py(49)create_connection()
-> LOG.debug('Connection: %s', conn)
(Pdb) n
> /usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/connection.py(50)create_connection()
-> LOG.debug('masakari client initialized: %s', conn.ha)
(Pdb) print LOG

(Pdb) n
ConfigException: ConfigEx...meters',)
> /usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/connection.py(50)create_connection()
-> LOG.debug('masakari client initialized: %s', conn.ha)
(Pdb) n
> /usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/connection.py(51)create_connection()
-> except Exception as e:
(Pdb) print e
*** NameError: name 'e' is not defined
(Pdb) n
> /usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/connection.py(52)create_connection()
-> raise e
(Pdb) print e
Problem with auth parameters
(Pdb)
____
From: Kwan, Louie
Sent: Friday, March 09, 2018 1:37 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [python-masakariclient] Installation issues

Two issues:

1.   Just download the latest and got some issues, cannot  ?

ubuntu@yow-tic-demo1:~$ masakari segment-list
('Problem with auth parameters', ', mode 'w' at 
0x7fe5188681e0>)


2.  Documentation: http://docs.openstack.org/developer/python-masakariclient

·Not Found

·The documentation is no long valid
Checked the bug list, it seems new issues? Any info will be much appreciated.

Thanks.

Louie

Note:
sudo su - stack
cd /home/stack
git clone https://github.com/openstack/python-masakariclient.git
cd python-masakariclient/
sudo python setup.py install
source ~/admin-openrc.sh
# To check the cli is working or not
masakari segment-list

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [python-masakariclient] Installation issues

[Kwan, Louie]

Two issues:

1.   Just download the latest and got some issues, cannot  ?

ubuntu@yow-tic-demo1:~$ masakari segment-list
('Problem with auth parameters', ', mode 'w' at 
0x7fe5188681e0>)


2.  Documentation: http://docs.openstack.org/developer/python-masakariclient

·Not Found

·The documentation is no long valid
Checked the bug list, it seems new issues? Any info will be much appreciated.

Thanks.

Louie

Note:
sudo su - stack
cd /home/stack
git clone https://github.com/openstack/python-masakariclient.git
cd python-masakariclient/
sudo python setup.py install
source ~/admin-openrc.sh
# To check the cli is working or not
masakari segment-list

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [masakari] [masakari-monitors] : Masakari notification failed.

[Kwan, Louie]

Good for now.

The issue should be related to some situations that VM stop instance task is 
taking longer and it seems one of the periodic task is timing out.

To avoid the exception, may  try to increase some of the timeout values. Or 
increase the looping interval for retry…

Thanks.
LK


From: Kwan, Louie
Sent: Tuesday, February 20, 2018 5:17 PM
To: 'OpenStack Development Mailing List (not for usage questions)'
Subject: [openstack-dev] [masakari] [masakari-monitors] : Masakari notification 
failed.

Hi Masakari community,

I would like to get your help  to understand what may be causing the Masakari 
notification failed.  I do get  success cases which the engine got the 
notification, VM got shutdown and rebooted ok.

Having said that, there are some cases that the notification failed and it 
seems there are some conflicts going on. 20% to 40% chance.

Feb 20 21:53:21 masakari-2 masakari-engine[3807]: 2018-02-20 21:53:21.517 
WARNING masakari.engine.drivers.taskflow.driver 
[req-ce909151-1afb-4f2f-abf4-f25d54f25c6b service None] Task 
'masakari.engine.drivers.taskflow.instance_failure.StopInstanceTask;instance:recovery'
 (e85dec06-1498-482c-a63a-51f855745c32) transitioned into state 'FAILURE' from 
state 'RUNNING'
Feb 20 21:53:21 masakari-2 masakari-engine[3807]: 1 predecessors (most recent 
first):
Feb 20 21:53:21 masakari-2 masakari-engine[3807]:   Flow 
'instance_recovery_engine': Conflict: Conflict

Is it normal that masakari notification would be failed because of timing or 
conflicting events? FYI, I only have one VM and one active notification.

Enclosed is the log file I got from the engine.

I do appreciate if anyone of you can provide some insight what to do with the 
failure. Any tip where to look at etc? Timeout?

Thanks.
Louie


| notification_uuid| generated_time | status   
| type | source_host_uuid | payload 

 |
+--++--+--+--+--+
| 42ccee84-0ea5-4163-84a5-028a0bb914a3 | 2018-02-20T21:52:03.00 | failed   
| VM   | 66c8b5b9-03f5-4843-8a9c-fa83af807a9b | {u'instance_uuid': 
u'565da9ba-3c0c-4087-83ca-32a5a1b00a55', u'vir_domain_event': 
u'STOPPED_FAILED', u'event': u'QEMU_GUEST_AGENT_ERROR'} |
| aa4184f3-b002-4ba8-a403-f22ccd4ce6b5 | 2018-02-20T21:42:54.00 | finished 
| VM   | 66c8b5b9-03f5-4843-8a9c-fa83af807a9b | {u'instance_uuid': 
u'565da9ba-3c0c-4087-83ca-32a5a1b00a55', u'vir_domain_event': 
u'STOPPED_FAILED', u'event': u'QEMU_GUEST_AGENT_ERROR'} |

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [libvrit] Can QEMU or LIBVIRT know VM is powering-off

[Kwan, Louie]

When turning off a VM by doing nova stop,  the Status and Task State is there 
for Nova. But can Libvirt / qemu programmatically figure out the 'Task State' 
that the VM is trying to powering-off ?.



For libvirt, it seems only know the "Power State"? Anyway to read the 
"powering-off" info?





+--+--++--+-++



| ID   | Name | Status | Task State   | Power 
State | Networks   |



+--+--++--+-++



| 09d65498-b1fe-4a99-9f43-4c365a79ff36 | c1   | ACTIVE | -| Running 
| public=172.24.4.6, 2001:db8::3 |



| 565da9ba-3c0c-4087-83ca-32a5a1b00a55 | iim1 | ACTIVE | powering-off | Running 
| public=172.24.4.5, 2001:db8::7 |



+--+--++--+-++



+--+--+-++-++



| ID   | Name | Status  | Task State | Power 
State | Networks   |



+--+--+-++-++



| 09d65498-b1fe-4a99-9f43-4c365a79ff36 | c1   | ACTIVE  | -  | Running  
   | public=172.24.4.6, 2001:db8::3 |



| 565da9ba-3c0c-4087-83ca-32a5a1b00a55 | iim1 | SHUTOFF | -  | Shutdown 
   | public=172.24.4.5, 2001:db8::7 |







Thanks.



Louie
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [masakari] [masakari-monitors] : Intrusive Instance Monitoring through QEMU Guest Agent Design Update

[Kwan, Louie]

Hi Sam

Make sense and will do option 2.

FYI, we are planning to upload the second iim patch within a week and may do 
the masakari engine in different patch shortly.

Thanks for your reply.

Louie

From: Sam P [mailto:sam47pr...@gmail.com]
Sent: Tuesday, February 20, 2018 3:33 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [masakari] [masakari-monitors] : Intrusive 
Instance Monitoring through QEMU Guest Agent Design Update

​Hi Louie,
 Thank you for patch and Sorry for the delay​ response.
I prefer ​option 2.
From Masakari point of view, this is an instance event. Because, even if some 
thing
wrong inside the VM, Masakari only can try to fix it by restart, rebuilt, 
migrate... etc the VM.
Which are the same recovery work flow for instance failures.  Therefore, I 
prefer option 2
rather than option1.
 Currently, we are discussing how to implement recovery method customization 
feature [0] in
Masakari. With this feature, you may able to call external workflows for 
certain failure events.
For this feature, different failure models required distinguishable events and 
option 3 will not
be appropriate.

[0] https://review.openstack.org/#/c/458023/


​> 1. define a new type of event for Intrusive Instance monitoring or
> 2. add a new event within the INSTANCE_EVENTS as we may  eventually integrate 
> with instance monitoring  or
>3.simply reuse the LIFECYCLE/STOPPED_FAILED event ( which is what we are 
>implementing for now.)

--- Regards,
Sampath


On Fri, Feb 16, 2018 at 12:05 AM, Kwan, Louie 
<louie.k...@windriver.com<mailto:louie.k...@windriver.com>> wrote:
We submitted the first implementation patch for the following blueprint

https://blueprints.launchpad.net/openstack/?searchtext=intrusive-instance-monitoring

i.e. https://review.openstack.org/#/c/534958/

The second patch  will be pushed within a week time or so.

One item we would like to seek clarification among the community is about  how 
we should integrate the notification within the masakari engine.

One option is to reuse what has been defined at  
masakari/engine/instance_events.py.

e.g.
def masakari_notifier(self, domain_uuid):
if self.getJournalObject(domain_uuid).getSentNotification():
LOG.debug('notifier.send_notification Skipped:' + domain_uuid)
else:
hostname = socket.gethostname()
noticeType = ec.EventConstants.TYPE_VM
current_time = timeutils.utcnow()
event = {
'notification': {
'type': noticeType,
'hostname': hostname,
'generated_time': current_time,
'payload': {
'event': 'LIFECYCLE',
'instance_uuid': domain_uuid,
'vir_domain_event': 'STOPPED_FAILED'
}
}
}
LOG.debug(str(event))
self.notifier.send_notification(CONF.callback.retry_max,
CONF.callback.retry_interval,
event)
self.getJournalObject(domain_uuid).setSentNotification(True)


​​
Should we


1.   define a new type of event for Intrusive Instance monitoring or

2.   add a new event within the INSTANCE_EVENTS as we may  eventually 
integrate with instance monitoring  or

3.   simply reuse the LIFECYCLE/STOPPED_FAILED event ( which is what we are 
implementing for now.)

One of our reference test case is to detect application meltdown within VM 
which QEMU may not  aware the failure. The recovery should pretty much be the 
same as LIFECYCLE/STOPPED_FAILED event. What do you think?

Thanks.
Louie

Ntoe:

Here is what we got from masakari/engine/instance_events.py

These are the events which needs to be processed by masakari in case of
instance recovery failure.
"""

INSTANCE_EVENTS = {
# Add more events and vir_domain_events here.
'LIFECYCLE': ['STOPPED_FAILED'],
'IO_ERROR': ['IO_ERROR_REPORT']
}


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [masakari] [notification api] How to clean up or purging of records

[Kwan, Louie]

Yeee.  Tha't is it.

Thanks
Louie

From: Bhor, Dinesh [dinesh.b...@nttdata.com]
Sent: Thursday, February 15, 2018 7:09 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [masakari] [notification api] How to clean up or 
purging of records

Hi Kwan Louie,

I think you are looking for this:
https://review.openstack.org/#/c/487430/

Thank you,
Dinesh Bhor


From: Kwan, Louie <louie.k...@windriver.com>
Sent: 16 February 2018 02:46:14
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [masakari] [notification api] How to clean up or 
purging of records

Hi All,

Just wondering, how can we clean up the masakari notification list or purging 
all old records in the DB?

openstack notification list
  returns too many old records

During semi-auto testing, I created a long list of history of records and would 
like to clean it up and avoid unnecessary actions.

Any short term solution is what I am looking for and/or ideas how to extend the 
CLI is also welcomed so that some of us can extend it later.

Thanks,
Louie


__
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [masakari] [notification api] How to clean up or purging of records

[Kwan, Louie]

Hi All,

Just wondering, how can we clean up the masakari notification list or purging 
all old records in the DB?

openstack notification list
  returns too many old records

During semi-auto testing, I created a long list of history of records and would 
like to clean it up and avoid unnecessary actions.

Any short term solution is what I am looking for and/or ideas how to extend the 
CLI is also welcomed so that some of us can extend it later.

Thanks,
Louie


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [automaton] How to extend automaton?

[Kwan, Louie]

Thanks for the reply. I will take the subclass approach for now.

It will be nice if we can dynamically register additional info or even a 
callback function  after building  a machine from a state space listing.

-LK

From: Joshua Harlow [harlo...@fastmail.com]
Sent: Wednesday, February 14, 2018 1:05 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [automaton] How to extend automaton?

As far a 1, I'd recommend just use functools.partial or make an object
with all the extra stuff u want and have that object provide a __call__
method.

As far as 2, you might have to subclass the FSM baseclass and add those
into the internal data-structure (same for 3 I think); ie this one @
https://github.com/openstack/automaton/blob/master/automaton/machines.py#L186-L191

Of course feel free to do it differently and submit a patch that folks
(myself and others) can review.

-Josh

Kwan, Louie wrote:
> https://github.com/openstack/automaton
>
> Friendly state machines for python.
>
> A few questions about automaton.
>
> 1.I would like to know can we addition parameters on on_enter or on_exit
> callbacks. Right now, it seems it only allows state and triggered_event.
>
> a.I have many FSM running for different objects and it is much easier if
> I can pass on the some sort of ID back to the callbacks.
>
> 2.Can we or how can we store extra attribute like last state change
> *timestamp*?
>
> 3.Can we store additional identify info for the FSM object? Would like
> to add an */UUID/*
>
> Thanks.
>
> Louie
>
> def print_on_enter(new_state, triggered_event):
>
> print("Entered '%s' due to '%s'" % (new_state, triggered_event))
>
> def print_on_exit(old_state, triggered_event):
>
> print("Exiting '%s' due to '%s'" % (old_state, triggered_event))
>
> # This will contain all the states and transitions that our machine will
>
> # allow, the format is relatively simple and designed to be easy to use.
>
> state_space = [
>
> {
>
> 'name': 'stopped',
>
> 'next_states': {
>
> # On event 'play' transition to the 'playing' state.
>
> 'play': 'playing',
>
> 'open_close': 'opened',
>
> 'stop': 'stopped',
>
> },
>
> 'on_enter': print_on_enter,
>
> 'on_exit': print_on_exit,
>
> },
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [masakari] [masakari-monitors] : Intrusive Instance Monitoring through QEMU Guest Agent Design Update

[Kwan, Louie]

We submitted the first implementation patch for the following blueprint

https://blueprints.launchpad.net/openstack/?searchtext=intrusive-instance-monitoring

i.e. https://review.openstack.org/#/c/534958/

The second patch  will be pushed within a week time or so.

One item we would like to seek clarification among the community is about  how 
we should integrate the notification within the masakari engine.

One option is to reuse what has been defined at  
masakari/engine/instance_events.py.

e.g.
def masakari_notifier(self, domain_uuid):
if self.getJournalObject(domain_uuid).getSentNotification():
LOG.debug('notifier.send_notification Skipped:' + domain_uuid)
else:
hostname = socket.gethostname()
noticeType = ec.EventConstants.TYPE_VM
current_time = timeutils.utcnow()
event = {
'notification': {
'type': noticeType,
'hostname': hostname,
'generated_time': current_time,
'payload': {
'event': 'LIFECYCLE',
'instance_uuid': domain_uuid,
'vir_domain_event': 'STOPPED_FAILED'
}
}
}
LOG.debug(str(event))
self.notifier.send_notification(CONF.callback.retry_max,
CONF.callback.retry_interval,
event)
self.getJournalObject(domain_uuid).setSentNotification(True)


Should we


1.   define a new type of event for Intrusive Instance monitoring or

2.   add a new event within the INSTANCE_EVENTS as we may  eventually 
integrate with instance monitoring  or

3.   simply reuse the LIFECYCLE/STOPPED_FAILED event ( which is what we are 
implementing for now.)

One of our reference test case is to detect application meltdown within VM 
which QEMU may not  aware the failure. The recovery should pretty much be the 
same as LIFECYCLE/STOPPED_FAILED event. What do you think?

Thanks.
Louie

Ntoe:

Here is what we got from masakari/engine/instance_events.py

These are the events which needs to be processed by masakari in case of
instance recovery failure.
"""

INSTANCE_EVENTS = {
# Add more events and vir_domain_events here.
'LIFECYCLE': ['STOPPED_FAILED'],
'IO_ERROR': ['IO_ERROR_REPORT']
}

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [automaton] How to extend automaton?

[Kwan, Louie]

https://github.com/openstack/automaton

Friendly state machines for python.



A few questions about automaton.



1.   I would like to know can we addition parameters on on_enter or on_exit 
callbacks. Right now, it  seems it  only allows  state and triggered_event.

a.   I have many FSM running for different objects and it is much easier if 
I can pass on the some sort of ID back to the callbacks.

2.   Can we or how can we store extra attribute like last state change 
timestamp?

3.   Can we store additional identify info for the FSM object? Would like 
to add an UUID





Thanks.

Louie



def print_on_enter(new_state, triggered_event):

   print("Entered '%s' due to '%s'" % (new_state, triggered_event))





def print_on_exit(old_state, triggered_event):

   print("Exiting '%s' due to '%s'" % (old_state, triggered_event))



# This will contain all the states and transitions that our machine will

# allow, the format is relatively simple and designed to be easy to use.

state_space = [

{

'name': 'stopped',

'next_states': {

# On event 'play' transition to the 'playing' state.

'play': 'playing',

'open_close': 'opened',

'stop': 'stopped',

},

'on_enter': print_on_enter,

'on_exit': print_on_exit,

},
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zuul] requirements-check FAILURE

[Kwan, Louie]

I got it now. Thanks all for the info. 

-Original Message-
From: Andreas Jaeger [mailto:a...@suse.com] 
Sent: Thursday, January 18, 2018 2:43 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Zuul] requirements-check FAILURE

On 2018-01-17 23:01,  Kwan, Louie  wrote:
> Would like to add the following module to openstack.masakari project
> 
> https://github.com/pytransitions/transitions
> 
> Got the following error with zuul requirements-check
> 
> Requirement set([Requirement(package=u'transitions', location='', 
> specifiers='>=0.6.4', markers=u'', comment='', extras=frozenset([]))]) not in 
> openstack/requirements
> 
> http://logs.openstack.org/88/534888/3/check/requirements-check/edec7bf/ara/
> 
> Any tip or insight to fix it?

Yes, read on how to add it:
https://docs.openstack.org/requirements/latest/

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] requirements-tox-validate-projects FAILURE

[Kwan, Louie]

Thanks Clark. Got the +1 from zuul.

By the way, I also changed to use openstack/automaton instead of transitions. 
It is all good now.

Louie

-Original Message-
From: Clark Boylan [mailto:cboy...@sapwetik.org] 
Sent: Thursday, January 18, 2018 5:28 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [requirements] requirements-tox-validate-projects 
FAILURE

On Thu, Jan 18, 2018, at 1:54 PM, Kwan, Louie wrote:
> Would like to add the following module to openstack.masakari project
> 
> https://github.com/pytransitions/transitions
> 
> https://review.openstack.org/#/c/534990/
> 
> requirements-tox-validate-projects failed:
> 
> http://logs.openstack.org/90/534990/6/check/requirements-tox-validate-projects/ed69273/ara/result/4ee4f7a1-456c-4b89-933a-fe282cf534a3/
> 
> What else need to be done?

Reading the log [0] the job failed because python-cratonclient removed its 
check-requirements job. This was done in 
https://review.openstack.org/#/c/535344/ as part of the craton retirement and 
should be fixed on the requirements side by 
https://review.openstack.org/#/c/535351/. I think a recheck at this point will 
come back green (so I have done that for you).

[0] 
http://logs.openstack.org/90/534990/6/check/requirements-tox-validate-projects/ed69273/job-output.txt.gz#_2018-01-18_20_07_54_531014

Hope this helps,
Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [requirements] requirements-tox-validate-projects FAILURE

[Kwan, Louie]

Would like to add the following module to openstack.masakari project

https://github.com/pytransitions/transitions

https://review.openstack.org/#/c/534990/

requirements-tox-validate-projects failed:

http://logs.openstack.org/90/534990/6/check/requirements-tox-validate-projects/ed69273/ara/result/4ee4f7a1-456c-4b89-933a-fe282cf534a3/

What else need to be done?

Thanks.
louie.k...@windriver.com


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Zuul] requirements-check FAILURE

[Kwan, Louie]

Would like to add the following module to openstack.masakari project

https://github.com/pytransitions/transitions

Got the following error with zuul requirements-check

Requirement set([Requirement(package=u'transitions', location='', 
specifiers='>=0.6.4', markers=u'', comment='', extras=frozenset([]))]) not in 
openstack/requirements

http://logs.openstack.org/88/534888/3/check/requirements-check/edec7bf/ara/

Any tip or insight to fix it?

Thanks.
louie.k...@windriver.com


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

[Kwan, Louie]

Hi Dinesh,

Thanks for the info.

'recovery_method' (choose from 'auto', 'reserved_host', 'auto_priority', 
'rh_priority')

What should be put for 

Same as the segment host

What should be put as   , it seems it 
accepts any arbitrary values.

Any more pointers. Much appreciated.

Thank you.
Louie Kwan 

From: Bhor, Dinesh [dinesh.b...@nttdata.com]
Sent: Wednesday, December 13, 2017 11:22 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance 
monitoring in devstack @ master

Hi Greg,


Looks like you don’t have “devstack-masakari” host registered in Masakari 
database.


Currently operator have to add all the hosts from failover-segment manually to 
Masakari database.


You can use below command:


Register failover-segment to Masakari database:

openstack segment create   


Register hosts under the created segment to Masakari database:

openstack segment host create



There is a work in progress on auto compute node registration.

Please refer: 
https://blueprints.launchpad.net/masakari-monitors/+spec/auto-compute-node-registration



Thank you,

Dinesh Bhor


From: Waines, Greg 
Sent: 13 December 2017 19:26:32
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance 
monitoring in devstack @ master


ok ... i changed all the domain related attributes in 
/etc/masakarimonitors/masakarimonitors.conf to be set to ‘default’ .

I seemed to get further,

but now get the following error when instancemonitor tries to send a 
notification:

  Bad Request (HTTP 400) (Request-ID: 
req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari 
could not be found.

however:

- devstack-masakari is in the /etc/hosts

- nova hypervisor-list shows devstack-masakari as a hypervisor hostname

- i am running both hostmonitor, processmonitor and instancemonitor



any ideas ?

see details below,

Greg.



2017-12-13 13:50:34.353 7637 INFO 
masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: 
type=VM, hostname=devstack-masakari, uuid=6ae0b09b-3e93-4f0c-b81b-fa140636f267, 
time=2017-12-13 13:50:34.353037, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-13 13:50:34.353 7637 INFO masakarimonitors.ha.masakari [-] Send a 
notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 
'payload': {'instance_uuid': '6ae0b09b-3e93-4f0c-b81b-fa140636f267', 
'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 
'generated_time': datetime.datetime(2017, 12, 13, 13, 50, 34, 353037)}}

2017-12-13 13:50:34.461 7637 WARNING masakarimonitors.ha.masakari [-] Retry 
sending a notification. (HttpException: Bad Request (HTTP 400) (Request-ID: 
req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari 
could not be found.): HttpException: HttpException: Bad Request (HTTP 400) 
(Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name 
devstack-masakari could not be found.

^C2017-12-13 13:50:42.462 7637 INFO oslo_service.service [-] Caught SIGINT 
signal, instantaneous exiting

root@devstack-masakari:~#

root@devstack-masakari:~# ping devstack-masakari

PING localhost (127.0.0.1) 56(84) bytes of data.

64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.049 ms

64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.022 ms

^C

--- localhost ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 999ms

rtt min/avg/max/mdev = 0.022/0.035/0.049/0.014 ms

root@devstack-masakari:~#



stack@devstack-masakari:~$ nova hypervisor-list

+--+-+---+-+

| ID   | Hypervisor hostname | State | Status  |

+--+-+---+-+

| 5fb1b09b-e5f5-465a-828a-2101135ff700 | devstack-masakari   | up| enabled |

+--+-+---+-+

stack@devstack-masakari:~$













From: Greg Waines 
Reply-To: "openstack-dev@lists.openstack.org" 

Date: Wednesday, December 13, 2017 at 8:17 AM
To: "openstack-dev@lists.openstack.org" 
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance 
monitoring in devstack @ master



So i believe the error is:

HttpException: HttpException: Expecting to find domain in project.



I have attached the /etc/masakarimonitors/masakarimonitors.conf file and the 
/etc/masakari/masakari.conf file .



See the domain related attributes in each file below.



Is the Default vs default causing this problem ?

Are there other domain related attributes that should be set to default ?

[openstack-dev] [all] How to use libxml2 with tox

[Kwan, Louie]

Would like to use libxml2 and having issues for tox.

What needs to be included in the requirements.txt file etc.

Any tip is much appreciated.

Thanks.
Louie
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev