In OSProfiler the current state is stored in thread-local variable. The definition of variable is in https://git.openstack.org/cgit/openstack/osprofiler/tree/osprofiler/profiler.py#n30.
It is observed however that the variable remains the same when a new green thread is forked. As result code https://git.openstack.org/cgit/openstack/nova/tree/nova/utils.py#n905 resets the variable of the process itself, not the one belonging to a thread. Steps to observe: 1. Print type and address of variable __local_ctx in osprofiler.profiler functions _Profiler.start and _Profiler.stop The output shows that the type of variable is 'thread._local' and address is the same across all invocations of Profiler's start and stop function. ** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1743586 Title: Nova operations fail when profiling is enabled Status in OpenStack Compute (nova): New Status in osprofiler: New Bug description: Nova server-related operations fail with HTTP 500 when executed with profiling. Nova server log contains error "IndexError: pop from an empty deque" and stack trace pointing to osprofiler code: ERROR nova.api.openstack File "/opt/stack/osprofiler/osprofiler/web.py", line 132, in __call__ ERROR nova.api.openstack return request.get_response(self.application) ERROR nova.api.openstack File "/opt/stack/osprofiler/osprofiler/profiler.py", line 348, in __exit__ ERROR nova.api.openstack stop() ERROR nova.api.openstack File "/opt/stack/osprofiler/osprofiler/profiler.py", line 87, in stop ERROR nova.api.openstack profiler.stop(info=info) ERROR nova.api.openstack File "/opt/stack/osprofiler/osprofiler/profiler.py", line 406, in stop ERROR nova.api.openstack self._notify("%s-stop" % self._name.pop(), info) Steps to reproduce: 0. Take the latest Devstack with osprofiler enabled. 1. Execute `openstack --os-profile SECRET_KEY server list` Observed behavior: $ openstack --os-profile SECRET_KEY server list The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-73de8073-94b7-44f0-96ca-5a0c5979636c) Trace ID: 5f0d7147-6e15-41c4-a61b-84f74e3787d4 Display trace with command: osprofiler trace show --html 5f0d7147-6e15-41c4-a61b-84f74e3787d4 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1743586/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
