Reviewed: https://review.opendev.org/751302 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7c1d964faab33a02fe2366b5194611252be045fc Submitter: Zuul Branch: master
commit 7c1d964faab33a02fe2366b5194611252be045fc Author: Artom Lifshitz <[email protected]> Date: Fri Sep 11 10:23:30 2020 -0400 Correctly disable greendns Previously, we were setting the environment variable to disable greendns in eventlet *after* import eventlet. This has no effect, as eventlet processes environment variables at import time. This patch moves the setting of EVENTLET_NO_GREENDNS before importing eventlet in order to correctly disable greendns. Closes-bug: 1895322 Change-Id: I4deed815c8984df095019a7f61d089f233f1fc66 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1895322 Title: Nova is not actually disabling greendns Status in OpenStack Compute (nova): Fix Released Bug description: Description =========== In [1], we began disabling greendns in eventlet to fix bug 1164822. This was done by setting the EVENTLET_NO_GREENDNS environment variable before importing eventlet. At import time, eventlet uses this env variable to enable/disable greendns [2]. Therefore, EVENTLET_NO_GREENDNS needs to be set before importing eventlet. Patch [3] changed that, setting the env var *after* importing eventlet, and thus re-enabling greendns in Nova. Steps to reproduce ================== ------------- Demonstration ------------- This is a bit of a hard one to reproduce, but there's a simple way to observe the necessity of setting the env var before importing eventlet: *** Setting the environment variable BEFORE the import *** --- Test script --- [artom@zoe scratchpad]$ cat eventlet-test.py import os import socket os.environ['EVENTLET_NO_GREENDNS'] = 'yes' import eventlet eventlet.monkey_patch() socket.gethostbyname('fake.local') --- Result: traceback DOES NOT include greendns.py (ie, environment variable worked) --- [artom@zoe scratchpad]$ python eventlet-test.py Traceback (most recent call last): File "eventlet-test.py", line 8, in <module> socket.gethostbyname('fake.local') socket.gaierror: [Errno -2] Name or service not known *** Setting the environment variable AFTER the import *** --- Test script --- [artom@zoe scratchpad]$ cat eventlet-test.py import os import socket import eventlet os.environ['EVENTLET_NO_GREENDNS'] = 'yes' eventlet.monkey_patch() socket.gethostbyname('fake.local') --- Result: traceback DOES include greendns.py (ie, environment variable DID NOT work) --- [artom@zoe scratchpad]$ python eventlet-test.py Traceback (most recent call last): File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 424, in resolve return _proxy.query(name, rdtype, raise_on_no_answer=raises, File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 380, in query return end() File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 359, in end raise result[1] File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 340, in step a = fun(*args, **kwargs) File "/home/artom/.local/lib/python3.8/site-packages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses) dns.resolver.NXDOMAIN: None of DNS query names exist: fake.local., fake.local.redhat.com. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "eventlet-test.py", line 8, in <module> socket.gethostbyname('fake.local') File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 550, in gethostbyname rrset = resolve(hostname) File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 434, in resolve raise EAI_NODATA_ERROR socket.gaierror: [Errno -2] Name or service not known ---------------- Real life impact ---------------- Downstream in our openstack product, the return of eventlet has been reported [4] as the cause of needless delays when failing over between Rabbit URLs in case one of the Rabbit servers goes down. In our deployment, DNS is not used, and IPv4 hostnames are just written to /etc/hosts. Looks like greendns tries to to IPv6 resolution regardless, needlessly using up 30 seconds in order to time out. Expected result =============== Almost-immediate failover to new Rabbit server. Actual result ============= greendns attempts IPv6 name resolution (though unclear of which Rabbit server - the failed one, or the next one), before finally, after 30 seconds, connecting to the next Rabbit server. Environment =========== This has been reported on stable/train, but should be the same on master. References ========== [1] https://review.opendev.org/#/c/26325/ [2] https://github.com/eventlet/eventlet/blob/v0.26.0/eventlet/green/socket.py#L20 [3] https://review.opendev.org/#/c/626952/ [4] https://bugzilla.redhat.com/show_bug.cgi?id=1860818 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1895322/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
