[issue11271] concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument
Brian Quinlan br...@sweetapp.com added the comment: On my crappy computer, ProcessPoolExecutor.map adds 3ms of added execution time per call using your test framework. What is your use case where that is too much? That being said, I don't have any objections to making improvements. If you want to pursue this, could you attach a working map_comprison.py? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11271 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11271] concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument
Tobias Brink tobias.br...@gmail.com added the comment: I can confirm an overhead of 2 ms to 3 ms using a relatively recent Intel Core i5 CPU. I (personally) believe these 3 ms to be a pretty big overhead on modern computers and I also believe that it would be relatively simple to reduce it. I don't have much time at the moment but I will try to produce a complete proof of concept patch for the futures module in the next weeks. I'd be happy to get some comments when it is finished. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11271 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11271] concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument
Changes by Ned Deily n...@acm.org: -- nosy: +bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11271 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11271] concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument
New submission from Tobias Brink tobias.br...@gmail.com: I tested the new concurrent.futures.ProcessPoolExecutor.map() in 3.2 with the is_prime() function from the documentation example. This was significantly slower than using multiprocessing.Pool.map(). Quick look at the source showed that multiprocessing sends the iterable in chunks to the worker process while futures sends always only one entry of the iterable to the worker. Functions like is_prime() which finish relatively fast make the communication overhead (at least I guess that is the culprit) very big in comparison. Attached is a file which demonstrates the problem and a quick workaround. The workaround uses the chunk idea from multiprocessing. The problem is that it requires the iterables passed to map() to have a length and be indexable with a slice. I believe this limitation could be worked around. -- components: Library (Lib) files: map_comparison.py messages: 128963 nosy: tbrink priority: normal severity: normal status: open title: concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument type: performance versions: Python 3.2 Added file: http://bugs.python.org/file20825/map_comparison.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11271 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11271] concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument
Tobias Brink tobias.br...@gmail.com added the comment: Playing around a bit I wrote the attached implementation which works with all iterables. -- Added file: http://bugs.python.org/file20826/new_processpoolexecutor.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11271 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
