Public bug reported: Throughout OpenStack, the eventlet library is used heavily for threading. In most cases, we use spawn_n() from eventlet which does not return whether the method that was wrapped with it raised an exception. Exceptions are instead just printed to stderr. Many deployers are using simple logging configurations that just log everything to a file. This poses a problem because spawn_n() does not use standard Python logging, but rather just prints a stack trace to stderr, like this: https://github.com/eventlet/eventlet/blob/master/eventlet/greenpool.py#L93
Any exceptions that get thrown in enventlet.spawn_n() are not shown in log files that are setup with normal log configurations that write to files. Deployers who have not been heavily involved with a lot of upstream development or do not know the internals of eventlet would think that there are no errors when tailing their log files which makes debugging problems extremely difficult. There are a few options to make error handling better 1. Use spawn() instead of spawn_n(). spawn() returns a greenthread object that would allow us to see the results of the call. We could check the results and log a proper exception if we used spawn(). The problem is, this would incur a performance penalty as spawn() is slower than spawn_n() 2. Expect deployers to write a custom log handler that will redirect stderr to their log file 3. Use proper exception handling inside of the functions that are called with spawn_n() so that exceptions we care about get caught and logged correctly A specific case where this happens and causes a problem is here: https://github.com/openstack/neutron/blob/7a722159f3bed10fbb26c4d0146f252da72ca0cd/neutron/services/segments/plugin.py#L222 In this scenario we reference aggregate.uuid. However if Nova API does not support microversion 2.41, then this will raise an AttributeError. However, the AttributeError will not be seen by most deployers because it will go to stderr rather than the log files setup with the Python logging framework. Then we start to see strange behavior everywhere and we have no idea why because our logs do not show anything. Logging and exception handling needs to be spruced up in this case and others so that eventlet does not devour important exceptions. Ultimately, we need to be much more cautious of exception handling inside of methods that are called through eventlet. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1759004 Title: No Errors or Warnings In Neutron When Nova API Microversions Are Not Supported Status in neutron: New Bug description: Throughout OpenStack, the eventlet library is used heavily for threading. In most cases, we use spawn_n() from eventlet which does not return whether the method that was wrapped with it raised an exception. Exceptions are instead just printed to stderr. Many deployers are using simple logging configurations that just log everything to a file. This poses a problem because spawn_n() does not use standard Python logging, but rather just prints a stack trace to stderr, like this: https://github.com/eventlet/eventlet/blob/master/eventlet/greenpool.py#L93 Any exceptions that get thrown in enventlet.spawn_n() are not shown in log files that are setup with normal log configurations that write to files. Deployers who have not been heavily involved with a lot of upstream development or do not know the internals of eventlet would think that there are no errors when tailing their log files which makes debugging problems extremely difficult. There are a few options to make error handling better 1. Use spawn() instead of spawn_n(). spawn() returns a greenthread object that would allow us to see the results of the call. We could check the results and log a proper exception if we used spawn(). The problem is, this would incur a performance penalty as spawn() is slower than spawn_n() 2. Expect deployers to write a custom log handler that will redirect stderr to their log file 3. Use proper exception handling inside of the functions that are called with spawn_n() so that exceptions we care about get caught and logged correctly A specific case where this happens and causes a problem is here: https://github.com/openstack/neutron/blob/7a722159f3bed10fbb26c4d0146f252da72ca0cd/neutron/services/segments/plugin.py#L222 In this scenario we reference aggregate.uuid. However if Nova API does not support microversion 2.41, then this will raise an AttributeError. However, the AttributeError will not be seen by most deployers because it will go to stderr rather than the log files setup with the Python logging framework. Then we start to see strange behavior everywhere and we have no idea why because our logs do not show anything. Logging and exception handling needs to be spruced up in this case and others so that eventlet does not devour important exceptions. Ultimately, we need to be much more cautious of exception handling inside of methods that are called through eventlet. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1759004/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
