Can someone comment on whether the following scenario has been discussed before or whether this is viewed by the community as a bug?
While debugging a couple of different issues our investigation has lead us down the path of needing to look at whether the oslo concurrency lock utilities are working properly or not. What we found is that it is possible for a greenthread to continuously acquire a lock even though there are other threads queued up waiting for the lock. For instance, a greenthread acquires a lock, does some work, releases the lock, and then needs to repeat this process over several iterations. While the first greenthread holds the lock other greenthreads come along and attempt to acquire the lock. Those subsequent greenthreads are added to the waiters list and suspended. The observed behavior is that as long as the first greenthread continues to run without ever yielding it will always re-acquire the lock even before any of the waiters. To illustrate my point I have included a short program that shows the effect of multiple threads contending for a lock with and without voluntarily yielding. The code follows, but the output from both sample runs are included here first. In both examples the output is formatted as "worker=XXX: YYY" where XXX is the worker number, and YYY is the number of times the worker has been executed while holding the lock. In the first example, notice that each worker gets to finish all of its tasks before any subsequence works gets to run even once. In the second example, notice that the workload is fair and each worker gets to hold the lock once before passing it on to the next in line. Example1 (without voluntarily yielding): ===== worker=0: 1 worker=0: 2 worker=0: 3 worker=0: 4 worker=1: 1 worker=1: 2 worker=1: 3 worker=1: 4 worker=2: 1 worker=2: 2 worker=2: 3 worker=2: 4 worker=3: 1 worker=3: 2 worker=3: 3 worker=3: 4 Example2 (with voluntarily yielding): ===== worker=0: 1 worker=1: 1 worker=2: 1 worker=3: 1 worker=0: 2 worker=1: 2 worker=2: 2 worker=3: 2 worker=0: 3 worker=1: 3 worker=2: 3 worker=3: 3 worker=0: 4 worker=1: 4 worker=2: 4 worker=3: 4 Code: ===== import eventlet eventlet.monkey_patch from oslo_concurrency import lockutils workers = {} synchronized = lockutils.synchronized_with_prefix('foo') @synchronized('bar') def do_work(index): global workers workers[index] = workers.get(index, 0) + 1 print "worker=%s: %s" % (index, workers[index]) def worker(index, nb_jobs, sleep): for x in xrange(0, nb_jobs): do_work(index) if sleep: eventlet.greenthread.sleep(0) # yield return index # hold the lock before starting workers to make sure that all worker queue up # on the lock before any of them actually get to run. @synchronized('bar') def start_work(pool, nb_workers=4, nb_jobs=4, sleep=False): for i in xrange(0, nb_workers): pool.spawn(worker, i, nb_jobs, sleep) print "Example1: sleep=False" workers = {} pool = eventlet.greenpool.GreenPool() start_work(pool) pool.waitall() print "Example2: sleep=True" workers = {} pool = eventlet.greenpool.GreenPool() start_work(pool, sleep=True) pool.waitall() Regards, Allain Allain Legacy, Software Developer, Wind River direct 613.270.2279 fax 613.492.7870 skype allain.legacy 350 Terry Fox Drive, Suite 200, Ottawa, Ontario, K2K 2W5 __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev