[issue16104] Use multiprocessing in compileall script
Claudiu.Popa added the comment: What can I do to move this forward? I believe all concerns have been addressed and it seems ready to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Claudiu.Popa added the comment: Here's a new patch which addresses Éric's last comments. Antoine, I don't have at my disposal a system without multiprocessing support. How does it crash? -- Added file: http://bugs.python.org/file34400/issue16104_5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Antoine Pitrou added the comment: Here's a new patch which addresses Éric's last comments. Antoine, I don't have at my disposal a system without multiprocessing support. How does it crash? Neither do I, but you will probably get an ImportError of some sort. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Claudiu.Popa added the comment: Here's a new version which catches ImportError for concurrent.futures and raises ValueError in `compile_dir` if `processes` was specified and concurrent.futures is unavailable. The only issue is that I don't know if this should be a ValueError or not. For instance, zipfile uses RuntimeError if `lzma` is unavailable. -- Added file: http://bugs.python.org/file34401/issue16104_6.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Changes by Claudiu.Popa pcmantic...@gmail.com: Added file: http://bugs.python.org/file34404/issue16104_7.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Claudiu.Popa added the comment: Thank you for the review, Éric! Here's the updated patch. -- Added file: http://bugs.python.org/file34368/issue16104_1.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Éric Araujo added the comment: FTR, py_compile and compileall use importlib in 3.4. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Changes by Claudiu.Popa pcmantic...@gmail.com: Added file: http://bugs.python.org/file34381/issue16104_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Éric Araujo added the comment: This looks ready to me. One thing: “make -j0” is the spelling for “run using all available cores”, whereas “compileall -j0” will use one process. I don’t know if this should be documented, changed or ignored. -- stage: patch review - commit review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Brett Cannon added the comment: I vote for changed so that -j0 uses all available cores as os.cpu_count() states. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Claudiu.Popa added the comment: I agree. I'll modify the patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Changes by Claudiu.Popa pcmantic...@gmail.com: Added file: http://bugs.python.org/file34383/issue16104_3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Éric Araujo added the comment: +if args.processes = 0: Is that correct? For make, I think I’ve always seen “-j0”, not negative values. Could you add a test for -j0? (i.e. check that “compileall -j0” calls the function with “processes=os.cpu_count()”) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Claudiu.Popa added the comment: regrtest does that, checking for j =0. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Claudiu.Popa added the comment: Here's a test for j0 == os.cpu_count. -- Added file: http://bugs.python.org/file34384/issue16104_4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Antoine Pitrou added the comment: Importing ProcessExecutor at the top-level means compileall will crash on systems which don't have multiprocessing support. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Éric Araujo added the comment: Patch looks good. Some comments on Rietveld. -- nosy: +eric.araujo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Changes by Éric Araujo mer...@netwok.org: -- stage: - patch review type: - enhancement versions: +Python 3.5 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Changes by Claudiu.Popa pcmantic...@gmail.com: Added file: http://bugs.python.org/file34339/issue16104.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Claudiu.Popa added the comment: Hello! Here's a draft patch. It adds a new *processes* parameter to *compile_dir* and a new command line parameter as well. -- keywords: +patch nosy: +Claudiu.Popa Added file: http://bugs.python.org/file33079/compileall_v1.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Changes by Brett Cannon br...@python.org: -- assignee: brett.cannon - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Changes by Brett Cannon br...@python.org: -- assignee: - brett.cannon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
New submission from Daniel Holth: compileall would benefit approximately linearly from additional CPU cores. There should be an option. The noisy output would have to change. Right now it prints compiling and then done synchronously with doing the actual work. -- messages: 171744 nosy: dholth priority: normal severity: normal status: open title: Use multiprocessing in compileall script ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Brett Cannon added the comment: This should probably use concurrent.futures instead of multiprocessing directly, but yes it would be useful. Then again, the whole module should probably be rewritten to use importlib as well. -- components: +Library (Lib) nosy: +brett.cannon priority: normal - low versions: +Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16104] Use multiprocessing in compileall script
Changes by Steven D'Aprano steve+pyt...@pearwood.info: -- nosy: +stevenjd ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16104 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
