Good for now.
The issue should be related to some situations that VM stop instance task is
taking longer and it seems one of the periodic task is timing out.
To avoid the exception, may try to increase some of the timeout values. Or
increase the looping interval for retry…
Thanks.
LK
From: Kwan, Louie
Sent: Tuesday, February 20, 2018 5:17 PM
To: 'OpenStack Development Mailing List (not for usage questions)'
Subject: [openstack-dev] [masakari] [masakari-monitors] : Masakari notification
failed.
Hi Masakari community,
I would like to get your help to understand what may be causing the Masakari
notification failed. I do get success cases which the engine got the
notification, VM got shutdown and rebooted ok.
Having said that, there are some cases that the notification failed and it
seems there are some conflicts going on. 20% to 40% chance.
Feb 20 21:53:21 masakari-2 masakari-engine[3807]: 2018-02-20 21:53:21.517
WARNING masakari.engine.drivers.taskflow.driver
[req-ce909151-1afb-4f2f-abf4-f25d54f25c6b service None] Task
'masakari.engine.drivers.taskflow.instance_failure.StopInstanceTask;instance:recovery'
(e85dec06-1498-482c-a63a-51f855745c32) transitioned into state 'FAILURE' from
state 'RUNNING'
Feb 20 21:53:21 masakari-2 masakari-engine[3807]: 1 predecessors (most recent
first):
Feb 20 21:53:21 masakari-2 masakari-engine[3807]: Flow
'instance_recovery_engine': Conflict: Conflict
Is it normal that masakari notification would be failed because of timing or
conflicting events? FYI, I only have one VM and one active notification.
Enclosed is the log file I got from the engine.
I do appreciate if anyone of you can provide some insight what to do with the
failure. Any tip where to look at etc? Timeout?
Thanks.
Louie
| notification_uuid | generated_time | status
| type | source_host_uuid | payload
|
+--------------------------------------+----------------------------+----------+------+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| 42ccee84-0ea5-4163-84a5-028a0bb914a3 | 2018-02-20T21:52:03.000000 | failed
| VM | 66c8b5b9-03f5-4843-8a9c-fa83af807a9b | {u'instance_uuid':
u'565da9ba-3c0c-4087-83ca-32a5a1b00a55', u'vir_domain_event':
u'STOPPED_FAILED', u'event': u'QEMU_GUEST_AGENT_ERROR'} |
| aa4184f3-b002-4ba8-a403-f22ccd4ce6b5 | 2018-02-20T21:42:54.000000 | finished
| VM | 66c8b5b9-03f5-4843-8a9c-fa83af807a9b | {u'instance_uuid':
u'565da9ba-3c0c-4087-83ca-32a5a1b00a55', u'vir_domain_event':
u'STOPPED_FAILED', u'event': u'QEMU_GUEST_AGENT_ERROR'} |
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
