[issue16104] Compileall script: add option to use multiple cores

[Gregory P. Smith]

Gregory P. Smith  added the comment:

This caused a regression in behavior.  compileall.compile_dir()'s ddir= 
parameter no longer does the right thing for any subdirectories.

https://bugs.python.org/issue39769

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Roundup Robot]

Roundup Robot added the comment:

New changeset 9efefcab817e by Brett Cannon in branch 'default':
Issue #16104: Allow compileall to do parallel bytecode compilation.
http://hg.python.org/cpython/rev/9efefcab817e

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Brett Cannon]

Brett Cannon added the comment:

Thanks for the patch, Claudiu!

--
resolution:  - fixed
stage: commit review - resolved
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Claudiu Popa]

Claudiu Popa added the comment:

Thank you for committing it. :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Claudiu Popa]

Claudiu Popa added the comment:

If there is nothing left to do for this patch, can it be committed?

--
Added file: http://bugs.python.org/file36590/16104.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Brett Cannon]

Changes by Brett Cannon br...@python.org:


--
assignee:  - brett.cannon

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Claudiu.Popa]

Claudiu.Popa added the comment:

Updated patch according to the python-dev thread:

- processes renamed to workers
- `workers` defaults to 1
- When `workers` is equal to 0, then `os.cpu_count` will be used
- When `workers`  1, multiple processes will be used
- When `workers` == 1, run normally (no multiple processes)
- Negative values raises a ValueError
- Will raise NotImplementedError if multiprocessing can't be used
(when `workers` equals to 0 or  1)

--
Added file: http://bugs.python.org/file35106/issue16104_12.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Jim Jewett]

Jim Jewett added the comment:

Trying to put bounds on the disagreements.  Does anyone disagree with any of 
the following:

(1)  compileall currently runs single-threaded in a single process.


(2)  This enhancement intends to allow parallelization by process.


(3)  Users MAY need to express whether they (require/forbid/are expressly 
apathetic concerning) paralellization.

(3A)  There is some doubt that this even needs to be user-controlled.

(3B)  If it is user-controlled, the patch proposes adding a processes 
parameter to do this.

(3C)  There have been suggestions of other names (notably workers), but *if* 
it is user-controlled, the idea of a new parameter is not controversial.


(4)  Users MAY need to control the degree of parallelization.

(4A)  If so, setting the value of the new parameter to a positive integer  1 
is an acceptable solution.

(4B)  There is not yet consensus on how to represent Use multi-processing, 
with the default degree for this system., Do NOT use multiprocessing., or I 
don't care.

(4C)  Suggested values have included 1, 0, -1, any negative number, None, and 
specific strings.  The precise mapping between some of these and the three 
cases of 4B is not agreed.


(5)  If multiprocessing is explicitly requested, what should happen when it is 
not available?

(5A)  Fall back to the current way, without multi-processing.

(5B)  Fall back to the current way, without multi-processing, but issue a 
Warning.

(5C)  Raise an Exception.  (ValueError, ImportError, NotImplemented?)


(6)  Portions of the documentation unrelated to this should be fixed.  But 
ideally, that would be done separately, and it will NOT be a pre-requisite to 
this patch.


---

Another potential value set

None (the default) == let the system parallelize as best it can -- as it does 
in multiprocessing.  If the system picks not in parallel at all, that is also 
OK, and no warning is raised.

0 == Do not parallelize.

positive integers == Use that many processes.

negative == ValueError

Would these uses of 0 and negative be too surprising for someone?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy:  -pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Claudiu.Popa]

Claudiu.Popa added the comment:

Added a new patch with improvements suggested by Jim. Thanks!

I removed the handling of processes=1, because it can still be useful: having a 
background worker which processes the files received from _walk_dir. Also, it 
checks that compile_dir receives a positive *processes* value, otherwise it 
raises a ValueError. As a side note, I just found that ProcessPoolExecutor / 
ThreadPoolExecutor don't verify the value of processes, leading to certain 
types of errors (see issue21362 for more details).
Jim, the default for processes is still None, meaning do not use multiple 
process, because the purpose of ProcessPoolExecutor makes it easy for it to 
use processes=None=os.cpu_count(). Here we want the user to be explicit about 
wanting multiple processes or not.

--
Added file: http://bugs.python.org/file35054/issue16104_9.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Steven D'Aprano]

Changes by Steven D'Aprano steve+pyt...@pearwood.info:


--
nosy:  -steven.daprano

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Claudiu.Popa]

Claudiu.Popa added the comment:

Add new patch with fixes proposed by Berker Peksag. Thanks for the review. 
Hopefully, this is the last iteration of this patch.

--
Added file: http://bugs.python.org/file35055/issue16104_10.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Claudiu.Popa]

Changes by Claudiu.Popa pcmantic...@gmail.com:


Added file: http://bugs.python.org/file35056/issue16104_11.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Jim Jewett]

Jim Jewett added the comment:

ProcessPoolExecutor already defaults to using cpu_count if max_workers is None. 
 Consistency with that might be useful too.  (and a default of 1 to mean 
nothing in parallel is sensible...)

--
nosy: +Jim.Jewett

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Claudiu.Popa]

Claudiu.Popa added the comment:

Added a new version of the patch which incorporates suggestions made by Jim. 
Thanks for the review!

--
Added file: http://bugs.python.org/file35018/issue16104_8.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16104] Compileall script: add option to use multiple cores

[Terry J. Reedy]

Changes by Terry J. Reedy tjre...@udel.edu:


--
title: Use multiprocessing in compileall script - Compileall script: add 
option to use multiple cores

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16104
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com