Hi,
I come across a questionable behavior of nova while I tried to use the
notify_on_api_faults configuration option [0] while testing the related
versioned notification transformation patch [1]. Based on the
description of the config option and the code that uses it [2] nova
sends and api.fault notification if the nova-api service encounters an
unhandle exception. There is a FaultWrapper class [3] added to the
pipeline of the REST request which catches every exception and calls
the notification sending.
Based on some debugging in devstack this FaultWrapper never catches any
exception. I injected a ValueError to the beginning of
nova.objects.aggregate.Aggregate.create method. This resulted in an
HTTPInternalServerError exception and HTTP 500 error code but the
exception handling part of the FaultWrapper [4] was never reached. So I
dig a bit deeper and I think I found the reason. Every REST API method
is decorated with expected_errors decorator [5] which as a last resort
translate the unexpected exception to HTTPInternalServerError. In the
wsgi stack the actual REST api call is guarded with
ResourceExceptionHandler context manager [7] which translates
HTTPException to a Fault [8]. Then Fault is catched and translated to
the REST response [7]. This way the exception never propagates back to
the FaultWrapper in [6] and therefore the api.fault notification is
never emitted.
You can see the api logs here [9] and the patch that I used to add the
extra traces here [10]. Please note that there is a compute.exception
notification visible in the log but that is a different notification
emitted from wrap_exception decorator [11] used in compute.manager [12]
and compute.api [13] only.
So my questions are:
1) Is it a bug in the nova wsgi or it is expected that the wsgi code
catches everything?
2) Do we need FaultWrapper at all if the wsgi stack catches every
exception?
3) Do we need api.fault notification at all? It seems nobody missed it
so far.
4) If we want to have api.fault notification then what would be the
good place to emit it? Maybe ResourceExceptionHandler at [8]?
I filed a bug for tracking purposes [14].
Cheers,
gibi
[0]
https://github.com/openstack/nova/blob/e611116e5822abf0e9f933cf6bd1b4c63007b170/nova/conf/notifications.py#L49
[1] https://review.openstack.org/#/c/469038
[2]
https://github.com/openstack/nova/blob/d68626595ed54698c7eb013a788ee3b98e068cdd/nova/notifications/base.py#L83
[3]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L79
[4]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L87
[5]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L325
[6]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L368
[7]
https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L637
[8]
https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L418
[9] https://pastebin.com/Eu6rBjNN
[10] https://pastebin.com/en4aFutc
[11]
https://github.com/openstack/nova/blob/master/nova/exception_wrapper.py#L57
[12]
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L105
[13]
https://github.com/openstack/nova/blob/master/nova/compute/api.py#L92
[14] https://bugs.launchpad.net/nova/+bug/1699115
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
