[issue20516] Concurrent.futures base concurrency improvement (with patch)
Changes by Glenn Langford glenn.langf...@gmail.com: -- nosy: -glangford ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20516 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13299] namedtuple row factory for sqlite3
Changes by Glenn Langford glenn.langf...@gmail.com: -- nosy: -glangford ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13299 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20369] concurrent.futures.wait() blocks forever when given duplicate Futures
Changes by Glenn Langford glenn.langf...@gmail.com: -- nosy: -glangford ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20369 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
Changes by Glenn Langford glenn.langf...@gmail.com: -- nosy: -glangford ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Changes by Glenn Langford glenn.langf...@gmail.com: -- nosy: -glangford ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: Any ideas how to debug this further? Wherever the cause of the problem might live, and to either work around it or gain additional information, here is one idea to consider. Do you need to submit your Futures just two at a time, and tightly loop every 15s? Why not submit a block of a larger number and wait for the block with as_completed(), logging for each completion. Then submit another block when they are all done. To control how many run at one time, create the Executor with max_workers=2 for example. (I had an app that ran 1,000 futures in this way, which worked fine). In general I suggest to only timeout when there is really a problem, not as an expected event. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: Under the hood, the behaviour of as_completed is quite different. So there is no guarantee it would behave the same. In any event, with millions of tasks you might consider Celery (I haven't used it myself): http://www.celeryproject.org -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: the wait is actually returning after the 15 seconds, although nothing is reported as finished...What kind of debug information from the futures would be useful? What is the state of the pending Futures that wait() is stuck on? (e.g. display f.running() and f.done() ). This could be logged any time the done set is empty after wait() returns. For each stuck Future, was it previously logged as completed by a prior call to wait()? What is the state of the ProcessPoolExecutor that the futures have been submitted to? Does it still function? (e.g. try submitting a trivial Future to the executor). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: My workload is launching two subprocess in parallel, and whenever one is ready, launches another one. Since you have timeout=15.0, wait() should return at least every 15s. Can you determine if the wait is being repeatedly called in the while loop, and if so what Futures it is waiting on? In other words, is wait() being called continuously, or is wait() called and never returns? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13299] namedtuple row factory for sqlite3
Glenn Langford added the comment: In abstract, I like the namedtuple interface for sqlite3 as well. One caution is that the approach suggested at http://peter-hoffmann.com/2010/python-sqlite-namedtuple-factory.html can have a dramatic impact on performance. For one DB-intensive application, I experienced 20+ seconds run time with the row factory (under 3.4), versus sub second without (identified with cProfile). Many thousands of calls to namedtuple_factory were not good. :) -- nosy: +glangford ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13299 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13299] namedtuple row factory for sqlite3
Glenn Langford added the comment: ...if I understand the proposed caching scheme, then repeated executions of the query SELECT a,b,c FROM table would result in cache hits, since the column names remain the same. I'm guessing this would resolve the performance problem in the app I saw, but it would be good to verify that performance is broadly similar with/without named tuples. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13299 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20566] asyncio as_completed() thrashes adding and removing callbacks
Glenn Langford added the comment: OK, code is ready for review at http://code.google.com/p/tulip/source/detail?r=674355412f33 This looks like a link to an old revision. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20566 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20516] Concurrent.futures base concurrency improvement (with patch)
Glenn Langford added the comment:
if the future is cancelled or finished, control is yielded back to the caller
with the condition's lock held.
Hmmm...I don't agree. I assume you are looking at this code:
with f._condition: # Lock the Future; yield if completed or add our Waiter
if f._state in [CANCELLED_AND_NOTIFIED, FINISHED]:
yield f
Note that the context manager will be called in this case to release the lock
before f is yielded to the caller.
class MiniContext():
def __init__(self):
pass
def __enter__(self):
print('Hello')
def __exit__(self, *args):
print('Goodbye')
def gen():
with MiniContext():
yield 1
print(next(gen()))
prints:
Hello
Goodbye
1
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20516
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20516] Concurrent.futures base concurrency improvement (with patch)
Glenn Langford added the comment: Actually, I think what you're seeing here is the context manager being garbage collected Yes indeed, I am with you now. Thanks for identifying the problem, and for your explanation. The future should not be yielded while locked. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20516 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20516] Concurrent.futures base concurrency improvement (with patch)
Glenn Langford added the comment: Updated patch: Fixed bug where quick yield in as_completed() did not release lock on future. -- Added file: http://bugs.python.org/file33999/futures-base-v2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20516 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20566] asyncio as_completed() thrashes adding and removing callbacks
New submission from Glenn Langford: In asyncio, tasks.py as_completed() appears to trigger adding and removing callbacks multiple times for the pending set of futures, each time a single future completes. For example, to wait for 5 futures which complete at different times: - as_completed() uses _wait() - _wait() will add 5 callbacks, one for each future - when one future completes, a callback is triggered and all 5 callbacks are removed; this is because _wait() was called with FIRST_COMPLETED - for the 4 remaining futures - 4 callbacks have to be added back again - when the next future completes, the 4 callbacks are removed etc… The worst case is if as_completed() is called to wait for all of a larger number of futures, which complete at different times. For example, with 100 futures worst case, ~10,000 callback adds and removes would be performed. (I am very new to the asyncio code, so I don't have a patch to offer at this point). -- components: Library (Lib) messages: 210679 nosy: glangford, gvanrossum, haypo, pitrou priority: normal severity: normal status: open title: asyncio as_completed() thrashes adding and removing callbacks type: performance versions: Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20566 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20566] asyncio as_completed() thrashes adding and removing callbacks
Changes by Glenn Langford glenn.langf...@gmail.com: Added file: http://bugs.python.org/file34000/test_thrash.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20566 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20566] asyncio as_completed() thrashes adding and removing callbacks
Changes by Glenn Langford glenn.langf...@gmail.com: Added file: http://bugs.python.org/file34001/output.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20566 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20566] asyncio as_completed() thrashes adding and removing callbacks
Glenn Langford added the comment: Glenn, what led you to discover this? It was found by code inspection. I recently started looking at concurrent.futures, really just curious as to how futures were implemented. Because one of the concurrent.futures bugs I raised also applied to asyncio, I started poking into the source for it as well. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20566 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20516] Concurrent.futures base concurrency improvement (with patch)
New submission from Glenn Langford: The current approach taken in as_completed() and wait() is to immediately lock the entire set of given Futures simultaneously. This limits concurrency, particularly when the number of futures is large, and also requires the complete set of Futures to be known up front. A different approach is to lock Futures one at a time, as they are given. In the case of as_completed(), completed futures can be yielded immediately. For completed futures, a waiter also does not have to be installed (whereas the current implementation installs waiters for all given Futures, regardless of state). A demonstration patch is attached which - locks Futures one at a time for as_completed() and all variations of wait() - reduces the overhead in tracking the state of each Future - makes it easier to add other user APIs if desired later - consolidates the machinery into a new internal class, reducing the amount of code -- components: Library (Lib) files: futures-base.patch keywords: patch messages: 210286 nosy: bquinlan, glangford, haypo, pitrou priority: normal severity: normal status: open title: Concurrent.futures base concurrency improvement (with patch) type: enhancement versions: Python 3.4, Python 3.5 Added file: http://bugs.python.org/file33920/futures-base.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20516 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: An idea for a different possible fix - rather than cleaning up waiters in wait() and as_completed(), could they be removed in Future.set_result() and Futures.set_exception() ? I'm not certain if any waiter should ever be notified twice; if not, perhaps set_result() and set_exception() could just include self._waiters = [] after all waiters have been signalled. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: Revised patch; I don't think there is a need to sort the keys when waiters are being removed since only one lock is acquired at a time. Stress tests on both wait() and as_completed() work with this approach. -- Added file: http://bugs.python.org/file33748/issue20319-no-sort.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20369] concurrent.futures.wait() blocks forever when given duplicate Futures
Glenn Langford added the comment: Updated patch with change to Doc/library/concurrent.futures.rst. -- Added file: http://bugs.python.org/file33751/issue20369.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20369 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: @Brian - Ah, I see now what you are referring to. The patch has changes to _create_and_install_waiters() which should not be there. The only code that needs to change is waiter removal as I originally suggested. I am set up with a dev environment now and will submit a revised patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: This patch shows the minimal desired outcome. It is not elegant in its current form, but does only what is necessary. Ultimately I think as_completed() should go its own way and not lock all Futures at once (#20297). -- Added file: http://bugs.python.org/file33725/issue20319.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20367] concurrent.futures.as_completed() fails when given duplicate Futures
Glenn Langford added the comment: Ah...ok, here is a patch that includes an update to Doc/library/concurrent.futures.rst as well. -- Added file: http://bugs.python.org/file33728/issue20367.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20367 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20367] concurrent.futures.as_completed() fails when given duplicate Futures
Glenn Langford added the comment: @Victor: Thank you, and I appreciate all your advice! I am still learning the dev environment but hope to be helpful. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20367 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20369] concurrent.futures.wait() blocks forever when given duplicate Futures
Glenn Langford added the comment: Updated patch with a test case, and added a minor note to the docstring to clarify behaviour. The use of sleep() in the test is not great, but it is the most obvious way to test and it is consistent with the approach used in other concurrent test cases. -- Added file: http://bugs.python.org/file33708/issue20369.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20369 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: It seems more plausible that the locks around the removals are fixing the bug but I don't see how. I'll look into it some more. It is the locks around the waiter removals that matter; I think there are only formatting changes elsewhere in the patch. The reason the locks make a difference is that there can be a race condition if multiple wait() calls compete to modify the f._waiters list. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20367] concurrent.futures.as_completed() fails when given duplicate Futures
Glenn Langford added the comment: Proposed patch for as_completed(). #20369 fixes wait(), and behaviour is consistent between the two. -- keywords: +patch Added file: http://bugs.python.org/file33684/issue20367.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20367 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20367] concurrent.futures.as_completed() fails when given duplicate Futures
Glenn Langford added the comment: Could you please try to write a unit test. Revised patch with unit test for as_completed(). -- Added file: http://bugs.python.org/file33685/issue20367.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20367 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20367] concurrent.futures.as_completed() fails when given duplicate Futures
Glenn Langford added the comment: Thanks for the feedback. The new patch is modified for PEP8 with naming consistent with other concurrent tests. assertEqual I think is clearer by checking list length, so it is not changed. The docstring is updated. I suggest asyncio be handled separately. -- Added file: http://bugs.python.org/file33686/issue20367.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20367 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
Changes by Glenn Langford glenn.langf...@gmail.com: -- nosy: +mark.dickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: @Victor: Would like to give a patch but I am not a core developer, and I don't version control set up yet. The proposed fix is based on reading the distribution source code. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
Glenn Langford added the comment: Comments on @Victor's comments - I will have a look at the dev guide. :-) I think you have identified another bug in the current code. Future._waiters is a list and so accept duplicated waiters. What this means is that the following code is broken, because it attempts to remove the Future twice from the pending set. for future in finished: yield future pending.remove(future) ## KeyError triggered on as_completed( [f,f] ) See attached test which demonstrates the breakage. Also, the behaviour of as_completed([f,f]) and wait([f,f], return_when=ALL_COMPLETED) is currently different. wait will only yield f once. -- Added file: http://bugs.python.org/file33656/test_dupfuture.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
Changes by Glenn Langford glenn.langf...@gmail.com: Removed file: http://bugs.python.org/file33656/test_dupfuture.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
Changes by Glenn Langford glenn.langf...@gmail.com: Added file: http://bugs.python.org/file33657/test_dupfuture.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20367] concurrent.futures.as_completed() fails when given duplicate Futures
Changes by Glenn Langford glenn.langf...@gmail.com: -- nosy: +haypo, mark.dickinson, tim.peters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20367 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20367] concurrent.futures.as_completed() fails when given duplicate Futures
New submission from Glenn Langford: concurrent.futures.as_completed([f,f]) will yield f twice, then fail with a KeyError for a Future f which is not completed. If the Future has already completed, as_completed([f,f]) will yield f once and does not trigger an exception. What is the correct behaviour? as_completed( [f,f] ) - yield f twice ? wait( [f,f], return_when=ALL_COMPLETED ) - yield f twice ? -- components: Library (Lib) files: test_dupfuture.py messages: 208952 nosy: glangford priority: normal severity: normal status: open title: concurrent.futures.as_completed() fails when given duplicate Futures type: behavior versions: Python 3.3, Python 3.4 Added file: http://bugs.python.org/file33658/test_dupfuture.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20367 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20367] concurrent.futures.as_completed() fails when given duplicate Futures
Glenn Langford added the comment: There is a subtlety in the as_completed() code which explains a lot - note that finished starts off as a set in the _AcquireFutures block. So if a Future f has already completed, as_completed( [f,f] ) will only yield f once, because f appears once in the finished set. Later on when waiter events are processed, finished turns into a list because of the line: finished = waiter.finished_futures So any duplicates in that list will cause problems in pending.remove(Future). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20367 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20369] concurrent.futures.wait() blocks forever when given duplicate Futures
New submission from Glenn Langford: For a Future f which has already completed, wait( [f,f], return_when=ALL_COMPLETED ) blocks forever. This is because the test in wait(): if len(done) == len(fs) is comparing the length of a set to the length of a list. If f has not completed, wait( [f,f] ) will yield f once. The behaviour should be consistent with as_completed() - see issue #20367. -- files: test_dupfuture_wait.py messages: 208960 nosy: glangford, haypo, mark.dickinson, tim.peters priority: normal severity: normal status: open title: concurrent.futures.wait() blocks forever when given duplicate Futures versions: Python 3.3, Python 3.4 Added file: http://bugs.python.org/file33660/test_dupfuture_wait.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20369 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
Glenn Langford added the comment: - you replaced the _AcquireFutures context manager on all futures with an individual lock. In my opinion, it should be changed in a different patch. I don't know which option is better, but if it is changed, it should be changed in the whole file. I can pursue the replacement of _AcquireFutures in a different patch, however I don't necessarily agree that it should be changed in the whole file. as_completed() is different than wait(), since wait() is obligated to return done and notdone Futures, which might require a different implementation. Initially, it may be better to only change as_completed(), where there is a clear concurrency win. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20369] concurrent.futures.wait() blocks forever when given duplicate Futures
Glenn Langford added the comment: Proposed patch...please treat with an appropriate level of suspicion since this is my first patch submission. :-) A corresponding change will be made to as_completed() for #20367. Suggestions welcome. -- keywords: +patch Added file: http://bugs.python.org/file33665/issue20369.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20369 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Glenn Langford added the comment: The same bug also exists in concurrent.futures.as_completed(). The minimal fix suggested here also works, but the bigger fix suggested in issue #20297 is recommended for as_completed(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
Glenn Langford added the comment: Uploading proposed new version of as_completed() for review. Note the following changes: - does not install waiters for Futures which are completed - locks only one Future at a time to improve concurrency (rather than locking all Futures at once); traverses Futures in the order given, as no need to sort into a canonical order - immediately yields each completed Future, without waiting to lock and examine other Futures - fixes locking bug in waiter removal -- Added file: http://bugs.python.org/file33590/as_completed_proposed.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
Changes by Glenn Langford glenn.langf...@gmail.com: -- nosy: +bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Changes by Glenn Langford glenn.langf...@gmail.com: -- nosy: +bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
New submission from Glenn Langford: concurrent.futures.wait() can get into a state where it blocks forever on waiter.event.wait(), even when the underlying Futures have completed. This is demonstrated in a stress test where a large number of wait() calls are run in multiple threads, contending for the same Futures. The cause is believed to be waiter removal, which is done without locking the Future. A suggested fix which appears to work is to change the following code in wait(): for f in fs: f._waiters.remove(waiter) to: for f in fs: with f._condition: f._waiters.remove(waiter) -- components: Library (Lib) files: stress_wait.py messages: 208592 nosy: glangford priority: normal severity: normal status: open title: concurrent.futures.wait() can block forever even if Futures have completed type: behavior versions: Python 3.3, Python 3.4 Added file: http://bugs.python.org/file33580/stress_wait.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
Glenn Langford added the comment: See also http://bugs.python.org/issue20319 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures
New submission from Glenn Langford: as_completed() calls _create_and_install_waiters() over all given Futures, even if some (or all) of the Futures have already completed. The following fragment of the as_completed code (from 3.3.3): with _AcquireFutures(fs): finished = set( f for f in fs if f._state in [CANCELLED_AND_NOTIFIED, FINISHED]) pending = set(fs) - finished waiter = _create_and_install_waiters(fs, _AS_COMPLETED) installs waiters into Futures contained in fs, rather than pending. A more efficient approach might be to only install waiters on the pending subset if necessary. This would be faster and would also result in less dwell time with all of the Future conditions locked via the _AcquireFutures context manager. Also: waiters are appended with the Future condition lock acquired, but are removed (at the conclusion of as_completed) without the _AcquireFutures condition lock. Is this correct? -- components: Library (Lib) messages: 208418 nosy: glangford priority: normal severity: normal status: open title: concurrent.futures.as_completed() installs waiters for already completed Futures type: performance versions: Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
