Reviewed: https://review.openstack.org/567506 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=20b3a04f1373633444c1064acca232f9f59cc133 Submitter: Zuul Branch: master
commit 20b3a04f1373633444c1064acca232f9f59cc133 Author: Radoslav Gerganov <[email protected]> Date: Thu May 10 11:20:01 2018 +0300 Add random sleep between retry calls to placement Attempt to reduce the resource provider contention by adding a random sleep between retry calls to placement. Change-Id: I4b7217f652dc2f5ff59f9d6a0178fa8f9325f706 Closes-Bug: #1770220 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1770220 Title: report client allocation retry handling insufficient Status in OpenStack Compute (nova): Fix Released Bug description: In stress testing of a nova+placement scenario where there is only one nova-compute process (and thus only one resource provider) but more than one thread worth of nova-scheduler it is fairly easy to trigger the "Failed scheduler client operation claim_resources: out of retries: Retry" error found near https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L110 (In a quick test on a devstack with a fake compute driver, 100 separate requests to boot one server, 13 failed for this reason.) If we imagine 4 threads: * A is one nova-scheduler * B is one placement request/response * C is another nova-scheduler * D is a different placement request/request A starts a PUT to /allocations, request B, at the start of which it reads the resource provider and gets a generation and the for whatever reason waits for a while. Then C starts a PUT to /allocations, request D, reads the same resource provider, same generation, but actually completes, getting to increment generation before B. When B gets to increment generation, it fails because now the generation it has is no good for the increment procedure. This is all working as expected but apparently is not ideal for high concurrency with low numbers of compute nodes. The currently retry loop has no sleep() and only counts up to 3 retries. It might make sense for it to do a random sleep before retrying (so as to introduce a bit of jitter in the system), and perhaps retry more times. Input desired. Thoughts? Another option, of course, is "don't run with so few compute nodes", but as we can likely expect this kind of stress testing (it was a real life stress test that worked fine in older (pre-claims-in-the- scheduler) versions that exposed this) we may wish to make it happier. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1770220/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
