[issue5309] distutils doesn't parallelize extension module compilation

[Giampaolo Rodola']

Giampaolo Rodola' added the comment:

I have 2 complaints about this:

1 - doc is missing: the only way to be aware of this is either by reading the 
3.6 what's new doc or by checking the cmdline helper

2 - -j "N" parameter could be optional: if not specified os.cpu_count() can be 
used.

--
nosy: +giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Roundup Robot]

Roundup Robot added the comment:

New changeset fc28c67fbbd3 by doko in branch 'default':
- Modules/Setup.dist: remove time extension duplicate, introduced by the fix 
for #5309.
https://hg.python.org/cpython/rev/fc28c67fbbd3

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Roundup Robot]

Roundup Robot added the comment:

New changeset 107669985805 by Berker Peksag in branch 'default':
Add whatsnew entry for issue #5309.
https://hg.python.org/cpython/rev/107669985805

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Berker Peksag]

Berker Peksag added the comment:

Here's a doc patch.

--
nosy: +berker.peksag
Added file: http://bugs.python.org/file37824/issue5309-doc.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Julian Taylor]

Julian Taylor added the comment:

very nice, thanks for adding this.

coincidentally numpy added the same to numpy.distutils independently just a 
week later, though numpy also accepts an environment variable to set the number 
of jobs.
This is useful for e.g. pip installations where one does not control the 
command line. Also an environment variable allows parallel jobs in environments 
where it is not guaranteed that the feature is available. E.g. you could just 
put it into your .bashrc and when building with 3.5 it will just work and 2.7 
will not fail.

Is the naming --parallel/j already fixed? I'll change the numpy options to the 
same name then.

Please also add it to the release notes so the feature can be discovered easier.

--
nosy: +jtaylor

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[STINNER Victor]

STINNER Victor added the comment:

It looks like compilation of Python 3.5 fails on FreeBSD 6.4 because of the 
changeset bbe57429eba0a9ec21fb0f1178f409f1bba44c22:

http://buildbot.python.org/all/builders/x86%20FreeBSD%206.4%203.x/builds/5061

Compile log:
---

building '_ctypes' extension
creating 
build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi
creating 
build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src
creating 
build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall 
-Wstrict-prototypes 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src 
-I./Include -I. -IInclude -I/usr/local/include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c 
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.c -o 
build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o
 -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall 
-Wstrict-prototypes 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src 
-I./Include -I. -IInclude -I/usr/local/include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c 
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.c -o 
build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o
 -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall 
-Wstrict-prototypes 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src 
-I./Include -I. -IInclude -I/usr/local/include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c 
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c -o 
build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o
 -Wall -fexceptions
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c: In 
function `py_dl_open':
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c:1336:
 warning: initialization discards qualifiers from pointer target type
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall 
-Wstrict-prototypes 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src 
-I./Include -I. -IInclude -I/usr/local/include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c 
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.c -o 
build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o
 -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall 
-Wstrict-prototypes 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src 
-I./Include -I. -IInclude -I/usr/local/include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c 
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.c -o 
build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o
 -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall 
-Wstrict-prototypes 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include 
-Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src 
-I./Include -I. -IInclude -I/usr/local/include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include 
-I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c 
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.c
 -o 

[issue5309] distutils doesn't parallelize extension module compilation

[STINNER Victor]

STINNER Victor added the comment:

There is a similar error on OpenIndiana buildbot, Python cannot be compiled 
since this build:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8664/steps/compile/logs/stdio

gcc -fPIC -Wsign-compare -g -O0 -Wall -Wstrict-prototypes 
-I/usr/local/include/ncursesw -m64 -Werror=declaration-after-statement 
-Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi/include 
-Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi 
-I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src
 -I./Include -I. -IInclude -I/usr/local/include 
-I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Include 
-I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build -c 
/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.S
 -o 
build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.o
 -Wall -fexceptions
gcc -fPIC -Wsign-compare -g -O0 -Wall -Wstrict-prototypes 
-I/usr/local/include/ncursesw -m64 -Werror=declaration-after-statement 
-Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi/include 
-Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi 
-I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src
 -I./Include -I. -IInclude -I/usr/local/include 
-I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Include 
-I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build -c 
/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/win32.S
 -o 
build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/win32.o
 -Wall -fexceptions
/var/tmp//ccqBa4n2.s: Assembler messages:
/var/tmp//ccqBa4n2.s:8: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:17: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:18: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:79: Error: suffix or operands invalid for `jmp'
/var/tmp//ccqBa4n2.s:138: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:147: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:155: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:156: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:163: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:176: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:210: Error: suffix or operands invalid for `jmp'
/var/tmp//ccqBa4n2.s:251: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:261: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:268: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:272: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:315: Error: suffix or operands invalid for `jmp'
/var/tmp//ccqBa4n2.s:358: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:359: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:370: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:383: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:416: Error: suffix or operands invalid for `jmp'
/var/tmp//ccqBa4n2.s:459: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:460: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:461: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:471: Error: suffix or operands invalid for `jmp'
*** WARNING: renaming _curses since importing it failed: ld.so.1: python: 
fatal: relocation error: file 
build/lib.solaris-2.11-i86pc.64bit-3.5-pydebug/_curses.so: symbol newscr: 
referenced symbol not found
*** WARNING: renaming _curses_panel since importing it failed: No module 
named '_curses'
error: [Errno 2] No such file or directory: 
'build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_decimal/_decimal.o'
make: *** [sharedmods] Error 1

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Antoine Pitrou]

Antoine Pitrou added the comment:

Thanks for noticing this.
Looking at an old build, the ctypes and curses compile errors under OpenIndiana 
are not new:
http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8658/steps/compile/logs/stdio

Simply, it seems that a ctypes compile error is now interpreted as a failure of 
the compilation build step. I'll try to fix that.

(the problem looks similar on the FreeBSD 6.4 buildbot)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[STINNER Victor]

STINNER Victor added the comment:

 the ctypes and curses compile errors under OpenIndiana are not new

FYI the compilation error of curses on OpenIndiana is *old*, at least 3 years 
old :-(
http://bugs.python.org/issue13552

I just opened the issue #22521 for the compilation error of ctypes on FreeBSD.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Jonas Wagner]

Jonas Wagner added the comment:

I don't understand where this error comes from...

The compilation commands are exactly the same in both the before and after 
logs. The order of commands is also the same.

The only difference is this message:

*** WARNING: renaming _ssl since importing it failed: 
build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ssl.so: Undefined symbol 
X509_check_ca

This message probably occurs because _ssl.so is now built in parallel with 
_ctypes.

In the logs, one can see that the command to build _ctypes.o is executed; 
because of that, I wonder why the file is missing. Maybe we could check for 
wrong dependency tracking in the _ctypes makefile (similar to 
http://bugs.python.org/issue22359)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Roundup Robot]

Roundup Robot added the comment:

New changeset 94af1af93670 by Antoine Pitrou in branch 'default':
Remove pointless vile hack that can cause the build step to fail when some 
extension modules can't be imported.
https://hg.python.org/cpython/rev/94af1af93670

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Antoine Pitrou]

Antoine Pitrou added the comment:

The change I just pushed should fix the failures, waiting for the buildbots to 
compile now.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Antoine Pitrou]

Antoine Pitrou added the comment:

Looks ok on OpenIndiana, closing now.

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Roundup Robot]

Roundup Robot added the comment:

New changeset bbe57429eba0 by Antoine Pitrou in branch 'default':
Issue #5309: distutils' build and build_ext commands now accept a ``-j``
https://hg.python.org/cpython/rev/bbe57429eba0

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Antoine Pitrou]

Antoine Pitrou added the comment:

This is now pushed.

--
resolution:  - fixed
stage: patch review - resolved
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Antoine Pitrou]

Antoine Pitrou added the comment:

Updated patch raising DistutilsOptionError. I couldn't find any docs so I 
didn't update them :-)

--
Added file: http://bugs.python.org/file36671/build_ext_parallel6.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Jonas Wagner]

Jonas Wagner added the comment:

I've checked the `dlopen` issue. This was due to a problem with 
UndefinedBehaviorSanitizer, and was solved by upgrading to Clang 3.5. It has 
little to do with this patch.

Using this patch and http://bugs.python.org/issue22359 , I now get reliable 
parallel builds.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Éric Araujo]

Éric Araujo added the comment:

 self.parallel = int(self.parallel) would raise ValueError when a
 non-number is passed. I suggest to print user-friendly error message.

The distutils idiom would be to catch the TypeError/ValueError and raise 
DistutilsOptionError.  Higher layers convert that to a message.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Antoine Pitrou]

Antoine Pitrou added the comment:

Updated patch:
- add the -j option to the build command as well (even though it's only used 
by build_ext)
- in setup.py, only force parallel compilation if make was called with -j

--
Added file: http://bugs.python.org/file36564/build_ext_parallel5.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
stage: needs patch - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Arfrever Frehtes Taifersar Arahesis]

Arfrever Frehtes Taifersar Arahesis added the comment:

- Is support for infinite jobs planned?
  (GNU make supports it for -j without number argument.)

- self.parallel = int(self.parallel) would raise ValueError when a non-number 
is passed. I suggest to print user-friendly error message.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Jonas Wagner]

Changes by Jonas Wagner jonas.wag...@epfl.ch:


--
nosy: +Sjlver

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Jonas Wagner]

Jonas Wagner added the comment:

Is there a reason this has not landed? The patch works perfectly for me, except 
for one issue:

@@ -268,6 +275,9 @@
 if self.undef:
 self.undef = self.undef.split(',')
 
+if self.parallel:
+self.parallel = int(self.parallel)
+
 if self.swig_opts is None:
 self.swig_opts = []
 else:

If self.parallel is True, this will set self.parallel to 1, causing it to use 
one worker instead of n (where n is the number of processors).

An updated patch is attached.

--
Added file: http://bugs.python.org/file36545/build_ext_parallel4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Stefan Behnel]

Stefan Behnel added the comment:

Yes, please. The sequential build in distutils is very annoying.

(too late for Py3.4, though, I guess...)

--
nosy: +scoder
versions: +Python 3.5 -Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Jonas Wagner]

Jonas Wagner added the comment:

With this patch, and on Ubuntu 14.04, occasionally modules fail to build with 
the following error:

*** WARNING: renaming _testbuffer since importing it failed: dlopen: cannot 
load any more object with static TLS

I'm not 100% sure if this is really due to the patch... but I've never seen it 
before. I'll try investigate a bit more.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Antoine Pitrou]

Antoine Pitrou added the comment:

Here is a working patch.

--
keywords: +patch
Added file: http://bugs.python.org/file32378/build_ext_parallel.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Antoine Pitrou]

Antoine Pitrou added the comment:

Here is an updated patch which only enables parallel building in setup.py.

--
Added file: http://bugs.python.org/file32380/build_ext_parallel2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Christian Heimes]

Christian Heimes added the comment:

Antoine's patch doesn't work for a fresh copy of Python

- add import error report to build_ext parallel (distutils doesn't use logging 
and doesn't have a working log.exception() function)
- make time a builtin module
- add hack for concurrent.futures.process
- add --parallel=i / -ji (jobs) argument to build_ext

--
Added file: http://bugs.python.org/file32382/build_ext_parallel3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5309] distutils doesn't parallelize extension module compilation

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever
title: setup.py doesn't parallelize extension module compilation - distutils 
doesn't parallelize extension module compilation

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5309
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com