[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Jason R. Coombs added the comment: I've flagged issue41207 as a regression stemming from this effort. -- nosy: +jaraco ___ Python tracker ___

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: I created bpo-39936 "Python fails to build _asyncio on module on AIX" to debug the AIX build failure. -- ___ Python tracker ___

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Michael Felt added the comment: After a checkout the build finishes (successfully) with: Python build finished successfully! The necessary bits to build these optional modules were not found: _gdbm _lzma _sqlite3 _tkinter _uuid

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Michael Felt added the comment: This was already confirmed by both bots... The last bot build that worked (and for both AIX bots) is: dffe4c07095e0c693e094d3c140e85a68bd8128e The first build that failed (for both) is: 1ec63b62035e73111e204a0e03b83503e1c58f2e See (pass)

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Change by Guido van Rossum : -- nosy: -gvanrossum ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: > OK. BEFORE I open a new issue, I am going to guess that something is wrong > with finding _socket.so - because it exists, but "something" is not finding > it. Did the build worked on AIX before the commit 1ec63b62035e73111e204a0e03b83503e1c58f2e which

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Michael Felt added the comment: I am confused. The bot complains nearly immediately about missing _socket Following modules built successfully but were removed because they could not be imported: _asyncio running build_scripts copying

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Michael Felt added the comment: OK. BEFORE I open a new issue, I am going to guess that something is wrong with finding _socket.so - because it exists, but "something" is not finding it. Following modules built successfully but were removed because they could not be imported:^M _asyncio

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Michael Felt added the comment: re: _socket - I'll start researching now, but it may still be related to this. a) if it looks like related to this issue I'll add a some report here b) in any case, as requested, I'll start a new issue - and leave it to you to decide to close that and continue

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: > At least a build will proceed, and I can look into this new, perhaps > unrelated issue re: socket later. > *** WARNING: renaming "_asyncio" since importing it failed: No module named > '_socket' Do you see any error when the _socket module is built?

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: New changeset add946855ad59c8f5c698aa0891d7e44f018 by Victor Stinner in branch 'master': bpo-39763: Add _bootsubprocess to build Python on AIX (GH-18872) https://github.com/python/cpython/commit/add946855ad59c8f5c698aa0891d7e44f018 --

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Michael Felt added the comment: Comes further. The build finishes, but socket and asyncio are missing... At least a build will proceed, and I can look into this new, perhaps unrelated issue re: socket later. *** WARNING: renaming "_asyncio" since importing it failed: No module named

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: > AttributeError: 'str' object has no attribute 'decode' Oops, it should now be fixed by my second commit. Please retry PR 18872. -- ___ Python tracker

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Michael Felt added the comment: Almost. The command is run, but not enough of the bootstrap is finished - it seems. File "/data/prj/python/git/python3-3.9/Lib/_aix_support.py", line 54, in _aix_bosmp64 out = out.decode("utf-8").strip().split(":") # type: ignore AttributeError: 'str'

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: > AttributeError: module 'subprocess' has no attribute 'check_output' I wrote PR 18872 to implement it for setup.py. Michael Felt: Can you please test it on AIX? -- ___ Python tracker

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Change by STINNER Victor : -- pull_requests: +18230 pull_request: https://github.com/python/cpython/pull/18872 ___ Python tracker ___

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Change by Michael Felt : -- pull_requests: +18204 pull_request: https://github.com/python/cpython/pull/18847 ___ Python tracker ___

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Michael Felt added the comment: The PR for 3.9 breaks AIX build (since https://buildbot.python.org/all/#/builders/119/builds/384) reporting the following: Traceback (most recent call last): File "/data/prj/python/python3-3.9/../git/python3-3.9/setup.py", line 69, in from

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: New changeset 00c77ae55a82548a6b45af73cdf712ea34910645 by Victor Stinner in branch 'master': bpo-39763: Refactor setup.py (GH-18778) https://github.com/python/cpython/commit/00c77ae55a82548a6b45af73cdf712ea34910645 --

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Change by STINNER Victor : -- pull_requests: +18134 pull_request: https://github.com/python/cpython/pull/18778 ___ Python tracker ___

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: New changeset 1ec63b62035e73111e204a0e03b83503e1c58f2e by Victor Stinner in branch 'master': bpo-39763: distutils.spawn now uses subprocess (GH-18743) https://github.com/python/cpython/commit/1ec63b62035e73111e204a0e03b83503e1c58f2e --

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Elad Lahav added the comment: "Attached fork_mt.py example uses exec_fn() function... which is not defined. Is it on purpose?" That was my mistake - a copy/paste of existing code from distutils. Since the example hangs before the script gets to exec_fn() I didn't notice the problem. Just

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: Attached fork_mt.py example uses exec_fn() function... which is not defined. Is it on purpose? -- ___ Python tracker ___

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: I wrote PR 18743 to reimplement distutils.spawn.spawn() function with the subprocess module. It also changes setup.py to use a basic implementation of the subprocess module if the subprocess module is not available: before required C extension modules are

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Change by STINNER Victor : -- keywords: +patch pull_requests: +18098 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18743 ___ Python tracker ___

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: > And setup.py does use use multiple processes via a thread pool, where each > thread calls fork(), if it detects that "make" was invoked with the parallel > (-j) option. Oh ok, now I get it :-) setup.py uses distutils to build extensions and

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Antoine Pitrou added the comment: @Elad: > I'm not convinced that a multi-threaded fork()+exec() from C would be any > better, unless the Python code goes to great lengths to avoid any > non-async-signal-safe operations between the fork() and the exec(). That's exactly what the C code in

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Change by Paul Ganssle : -- nosy: -p-ganssle ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Elad Lahav added the comment: "setup.py doesn't use multiprocessing. multiprocessing is super complex. Would it be possible to write a reproducer which doesn't use multiprocessing?" But the problem is with the handling of fork() by Python modules, and specifically with multi-threaded

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

STINNER Victor added the comment: > Note the original problem stems when you using "setup.py -j ". setup.py doesn't use multiprocessing. multiprocessing is super complex. Would it be possible to write a reproducer which doesn't use multiprocessing? > Stack trace: (...) Can you try to get

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Elad Lahav added the comment: I may be missing something, but subprocess depends on _posixsubprocess, which doesn't get built until setup.py is run, which results in a circular dependency. -- ___ Python tracker

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Elad Lahav added the comment: As sure as I can be given my limited experience debugging Python... Luckily I do know my way around the QNX kernel ;-) The stack trace for the child shows it stuck on a semaphore with a count value of 0. A print in the logging module shows that the child gets

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Guido van Rossum added the comment: And you're sure that your "toy" example hangs in the at-fork handler. In that case I can't help you further, I don't have the resources or time to try out QNX. Your best bet is indeed to change distutils.spawn to not use fork+exec -- do you think you can

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Elad Lahav added the comment: OK, but that's not the problem I see. The parent calls fork(), creates a child that then runs the atfork() handlers *before* returning from the os.fork() call (which is the expected behaviour). At least one of those atfork() handlers is the one registered by

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Guido van Rossum added the comment: The key is that os.execl() never returns. My understanding is that the hang happens when the child code returns (or raises) in the forked child process, because the semaphore state is cloned by the fork. There is no logging here. I just added some

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Elad Lahav added the comment: "When I uncomment the os.execl() line, the program runs and completes." In that case I'm not sure it is the same issue. The child processes in your case executed their part of the new_process function, which then returned. Nevertheless from the trace it looks

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Guido van Rossum added the comment: When I run this on macOS, either with the bug, or with the buggy line commented out, I get the same hang. When I interrupt it, I get a separate traceback from each thread. Here's the full session: Parent 78918 Parent 78919 Child 78918 Parent 78920 Child

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Elad Lahav added the comment: "Your script contains a bug (there is no definition of 'exec_fn')" Told you I wasn't much of a Python developer ;-) This was just copy-pasted from spawn.py and I missed the fact that exec_fn() is not a library function. Your last comment wasn't clear - do you

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Guido van Rossum added the comment: @Antoine: I can sort of see why you consider the small example a red herring, since it mixes threads and fork. Nevertheless, a simpler version (not using an executor) would probably work just fine? It's the edge cases where the mixture becomes explosive.

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Elad Lahav added the comment: I'm not convinced that a multi-threaded fork()+exec() from C would be any better, unless the Python code goes to great lengths to avoid any non-async-signal-safe operations between the fork() and the exec(). So along with the proposed change to switch to the

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Change by Antoine Pitrou : -- nosy: +dstufft, eric.araujo, p-ganssle, vstinner ___ Python tracker ___ ___ Python-bugs-list mailing

[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)

Change by Antoine Pitrou : -- title: Hang after fork due to logging trying to reacquire the module lock in an atfork() handler -> distutils.spawn should use subprocess (hang in parallel builds on QNX) ___ Python tracker