[issue33725] Python crashes on macOS after fork with no exec

[Eryk Sun]

Change by Eryk Sun :


--
components: +macOS -Library (Lib)
nosy: +ned.deily, ronaldoussoren -ahmedsayeed1982
versions: +Python 3.8 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Eryk Sun]

Change by Eryk Sun :


--
Removed message: https://bugs.python.org/msg405703

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ahmed Sayeed]

Ahmed Sayeed  added the comment:

$ ./gdb -nx -q --data-directory=data-directory ~/a.out -ex "set confirm off" 
-ex "file -readnow ~/a.out"  http://www-look-4.com/property/houses-in-france/
Reading symbols from /home/simark/a.out...
Reading symbols from ~/a.out... 
https://komiya-dental.com/crypto/new-coins/
/home/simark/src/binutils-gdb/gdb/dwarf2/read.c:8098: internal-error: 
void create_all_comp_units(dwarf2_per_objfile*): Assertion 
`per_objfile->per_bfd- 
http://www.iu-bloomington.com/crypto/china-affect-on-crypto/ 
>all_comp_units.empty ()' failed. 

This is a recurring problem that exposes a design issue 
https://waytowhatsnext.com/crypto/cryptocurrency-taxes/ in the DWARF
per-BFD sharing feature.  Things work well when loading a binary with 
https://www.mktrade.fi/muottivalmistus
the same method (with/without index, with/without readnow) twice 
http://www.wearelondonmade.com/property/cars-as-house/ in a
row.  But they don't work so well when loading a binary with different
methods.  See this previous fix, for example: 
http://www.jopspeech.com/property/slim-pen-2/

efb763a5ea35 ("gdb: check for partial symtab presence in 
dwarf2_initialize_objfile") http://joerg.li/services/kia-rio-price/

That one handled the case where the first load is normal (uses partial
symbols) and the second load uses an index. 
http://connstr.net/services/mobile-games/

The problem is that when loading an objfile with a method A, we create a 
dwarf2_per_bfd and some dwarf2_per_cu_data and initialize them with the 
http://embermanchester.uk/property/chat-themes/ 
data belonging to that method.  When loading another obfile sharing the
same BFD but with a different method B, it's not 
http://www.slipstone.co.uk/services/chevrolet-bolt/ clear how to re-use the
dwarf2_per_bfd/dwarf2_per_cu_data previously created, because they
contain the data specific to method A. 
http://www.logoarts.co.uk/services/affordable-printer/

I think the most sensible fix would be to not share a dwarf2_per_bfd
between two objfiles loaded with different methods.  That means that two
objfiles sharing the same BFD 
http://www.acpirateradio.co.uk/tech/forest-fires/ and loaded the same way would 
share a
dwarf2_per_bfd.  Two objfiles sharing the same BFD but loaded with
different methods would use two different dwarf2_per_bfd structures.
http://www.compilatori.com/services/whatsapp-service/
However, this isn't a trivial change.  So to fix the known issue quickly
(including in the gdb 10 branch), this patch just disables all
dwarf2_per_bfd sharing for objfiles using READNOW.

Generalize the gdb.base/index-cache-load-twice.exp test to test all
the possible combinations of loading a file with partial symtabs, index
and readnow.  Move it to gdb.dwarf2, 
https://www.webb-dev.co.uk/crypto/crypto-fell/ since it really exercises 
features
of the DWARF reader.

--
components: +Library (Lib) -macOS
nosy: +ahmedsayeed1982 -Ryan May, barry, blalterman, davin, gregory.p.smith, 
josh.r, kapilt, lukasz.langa, mark.dickinson, miss-islington, mouse07410, 
ned.deily, pablogsal, pitrou, ronaldoussoren, tdsmith, terry.reedy
versions: +Python 3.7 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I don't think there's really anything more to do here.  I'm closing the issue.  
Let's open a new one if needed at some future point.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Since the default has been different on different systems for as long as I 
remember, I see no reason to break code on *nix in the name of 'consistency'.  
Asyncio also works different on different systems.

Aside from that idea, is there anything else left for this issue?  Especially 
at this time as opposed to some possible future when macOS changes?

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[B. L. Alterman]

Change by B. L. Alterman :


--
nosy: +blalterman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Mouse]

Mouse  added the comment:

@mark.dickinson, thank you. Following your suggestion, I've added a comment in 
#28965, and created a new issue https://bugs.python.org/issue40106.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Mark Dickinson]

Mark Dickinson  added the comment:

@Mouse: replace the last block of your code with something like this:

# Start worker processes
workers = []
for m in range(nProcesses):
p = mp.Process(target=getElements, args=(q, tSleep / 1000, m))
workers.append(p)
p.start()

# Wait for all workers to complete
for p in workers:
p.join()

But I don't think this tracker issue is the right place to have this 
conversation. It's highly unlikely that this change will be reverted - there 
are strong reasons to avoid fork on macOS, and the issue you're reporting isn't 
directly related to this one: it's an issue with using "spawn" on any OS.

However, there may be scope for improving the documentation so that fewer users 
fall into this trap. I'd suggest opening another issue for that, or continuing 
the conversation on #28965.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Mouse]

Mouse  added the comment:

Also, adding `p.join()` immediately after `p.start()` in my sample code showed 
this timing:
```
$ time python3.8 multi1.py 
worker 0 done, got 100 numbers
worker 1 done, got 0 numbers
worker 2 done, got 0 numbers
worker 3 done, got 0 numbers

real0m2.342s
user0m0.227s
sys 0m0.111s
$ 
```

Setting instead start to `fork` showed this timing:
```
$ time python3.8 multi1.py 
worker 2 done, got 25 numbers
worker 0 done, got 25 numbers
worker 1 done, got 25 numbers
worker 3 done, got 25 numbers

real0m0.537s
user0m0.064s
sys 0m0.040s
$ 
```

The proposed fix is roughly four times slower, compared to reverting start to 
`fork`.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Mouse]

Mouse  added the comment:

@mark.dickinson, the issue you referred to did not show a working sample. Could 
you demonstrate on my example how it should be applied? Thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Mark Dickinson]

Mark Dickinson  added the comment:

@Mouse: see #28965. The fix for the code you show is to join the child 
processes before the main process starts exiting.

--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Mouse]

Mouse  added the comment:

Tried 'spawn', 'fork', 'forkserver'. 

- 'spawn' causes consistent `FileNotFoundError: [Errno 2] No such file or 
directory`;
- 'fork' consistently works (tested on machines with 4 and 20 cores);
- 'forkserver' causes roughly half of the processes to crash with 
`FileNotFoundError`, the other half succeeds (weird!).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Mouse]

Mouse  added the comment:

The fix applied for this problem actually broke multiprocessing on MacOS. The 
change to the new default 'spawn' from 'fork' causes program to crash in 
spawn.py with `FileNotFoundError: [Errno 2] No such file or directory`.

I've tested this on MacOS Catalina 10.15.3 and 10.15.4, with Python-3.8.2 and 
Python-3.7.7.

With Python-3.7.7 everything works as expected.

Here's the output:
{{{
$ python3.8 multi1.py 
Traceback (most recent call last):
  File "", line 1, in 
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py",
 line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py",
 line 126, in _main
self = reduction.pickle.load(from_parent)
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/synchronize.py",
 line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
}}}

Here's the program:
{{{
#!/usr/bin/env python3
#
# Test "multiprocessing" package included with Python-3.6+
#
# Usage:
#./mylti1.py [nElements [nProcesses [tSleep]]]
#
#nElements  - total number of integers to put in the queue
# default: 100
#nProcesses - total number of parallel processes/threads
# default: number of physical cores available
#tSleep - number of milliseconds for a thread to sleep
# after it retrieved an element from the queue
# default: 17
#
# Algorithm:
#   1. Creates a queue and adds nElements integers to it,
#   2. Creates nProcesses threads
#   3. Each thread extracts an element from the queue and sleeps for tSleep 
milliseconds
#

import sys, queue, time
import multiprocessing as mp


def getElements(q, tSleep, idx):
l = []  # list of pulled numbers
while True:
try:
l.append(q.get(True, .001))
time.sleep(tSleep)
except queue.Empty:
if q.empty():
print(f'worker {idx} done, got {len(l)} numbers')
return


if __name__ == '__main__':
nElements = int(sys.argv[1]) if len(sys.argv) > 1 else 100
nProcesses = int(sys.argv[2]) if len(sys.argv) > 2 else mp.cpu_count()
tSleep = float(sys.argv[3]) if len(sys.argv) > 3 else 17

# Uncomment the following line to make it working with Python-3.8+
#mp.set_start_method('fork')

# Fill the queue with numbers from 0 to nElements
q = mp.Queue()
for k in range(nElements):
q.put(k)

# Start worker processes
for m in range(nProcesses):
p = mp.Process(target=getElements, args=(q, tSleep / 1000, m))
p.start()
}}}

--
nosy: +mouse07410
type:  -> crash

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ryan May]

Change by Ryan May :


--
nosy: +Ryan May

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:

As far as I can tell, the only thing left to do for this issue is to add a 
documentation warning to the 3.7 documents similar to what was added to 3.8 but 
without the change in default.  A PR would be nice.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[miss-islington]

miss-islington  added the comment:


New changeset d74438b633184bbd8d775d7118d6f12f6f825a96 by Miss Islington (bot) 
in branch '3.8':
bpo-33725, multiprocessing doc: rephase warning against fork on macOS (GH-13841)
https://github.com/python/cpython/commit/d74438b633184bbd8d775d7118d6f12f6f825a96


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[miss-islington]

Change by miss-islington :


--
pull_requests: +13726
pull_request: https://github.com/python/cpython/pull/13849

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 1e77ab0a35cf95318bb4893f7253a30f73201163 by Victor Stinner in 
branch 'master':
bpo-33725, multiprocessing doc: rephase warning against fork on macOS (GH-13841)
https://github.com/python/cpython/commit/1e77ab0a35cf95318bb4893f7253a30f73201163


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

> Thanks Victor.  I don’t think “reliable” is strong enough, since this will 
> definitely lead to core dumps under certain conditions.  What about: (...)

That sounds better: I wrote PR 13841.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13718
pull_request: https://github.com/python/cpython/pull/13841

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On Jun 4, 2019, at 08:11, STINNER Victor  wrote:
> Ned Deily:
>> No, it has *always* been unsafe. What's new as of 10.13/14 is that macOS 
>> tries much harder at runtime to detect such cases and more predictably cause 
>> an error rather than letter than let the process run on and possibly fail 
>> nondeterministically.
> 
> Hum, in the doc, I wrote:
> 
> .. versionchanged:: 3.8
> 
>   On macOS, *spawn* start method is now the default: *fork* start method is no
>   longer reliable on macOS, see :issue:`33725`.
> 
> Should we change this text? Any proposition?

Thanks Victor.  I don’t think “reliable” is strong enough, since this will 
definitely lead to core dumps under certain conditions.  What about:

   On macOS, the *spawn* start method is now the default.  The *fork* start 
method should
   be considered unsafe as it can lead to crashes of the subprocess.  See 
:issue:`33725`.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

Ned Deily:
> No, it has *always* been unsafe. What's new as of 10.13/14 is that macOS 
> tries much harder at runtime to detect such cases and more predictably cause 
> an error rather than letter than let the process run on and possibly fail 
> nondeterministically.

Hum, in the doc, I wrote:

.. versionchanged:: 3.8

   On macOS, *spawn* start method is now the default: *fork* start method is no
   longer reliable on macOS, see :issue:`33725`.

Should we change this text? Any proposition?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On May 28, 2019, at 17:38, Ned Deily  wrote:
> 
> Ned Deily  added the comment:
> 
>> To be clear, what is unsafe on macOS (as of 10.13, but even more so on 
>> 10.14) is calling into the Objective-C runtime between fork and exec.
> 
> No, it has *always* been unsafe. What's new as of 10.13/14 is that macOS 
> tries much harder at runtime to detect such cases and more predictably cause 
> an error rather than letter than let the process run on and possibly fail 
> nondeterministically.

Right, thanks for the additional nuance.  I think what changed is that in 
10.13, Apple added a warning output when this condition occurred, and in 10.14 
they actually abort the subprocess.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On May 28, 2019, at 17:21, STINNER Victor  wrote:
> 
> 
> STINNER Victor  added the comment:
> 
>> To be clear, what is unsafe on macOS (as of 10.13, but even more so on 
>> 10.14) is calling into the Objective-C runtime between fork and exec.  The 
>> problem for Python is that it’s way too easy to do that implicitly, thus 
>> causing the macOS to abort the subprocess in surprising ways.
> 
> Do only a few Python module use the Objective-C runtime? Or is it basically 
> "everything"?
> 
> If it's just a few, would it be possible to emit a warning or even an 
> exception if called in a child process after fork?

I think it’s hard to know, but I found it through a path that lead from 
requests to _scproxy.c.  Here’s everything I know about the subject:

https://wefearchange.org/2018/11/forkmacos.rst.html

So yes, it’s theoretically possible to do *some* between fork and exec and not 
crash, and it’s of course perfectly safe to call exec pretty much right after 
fork.  It’s just hard to know for sure, and there are surprising ways to get 
into the Objective-C runtime.

I think we won’t be able to work around all of Apple’s choices here.  
Documentation is the best way to handle it in <=3.7, and changing the default 
makes sense to me for 3.8.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Change by Ned Deily :


--
Removed message: https://bugs.python.org/msg343843

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:

> To be clear, what is unsafe on macOS (as of 10.13, but even more so on 10.14) 
> is calling into the Objective-C runtime between fork and exec.

No, it has *always* been unsafe. What's new as of 10.13/14 is that macOS tries 
much harder at runtime to detect such cases and more predictably cause an error 
rather than let the process run on and possibly fail nondeterministically. 

> Do only a few Python module use the Objective-C runtime? Or is it basically 
> "everything"?

I don't think we should try to second-guess this.  We now recognize that using 
fork like this on macOS has always been dangerous.  For some programs it will 
be fine, for others it won't.  People have had many macOS and Python releases 
to deal with this; if it works for their application, we shouldn't be changing 
the default for them.  But let's make it easier for new users to do the right 
thing - first by documenting the pitfall, then, in 3.8, changing the default.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:

> To be clear, what is unsafe on macOS (as of 10.13, but even more so on 10.14) 
> is calling into the Objective-C runtime between fork and exec.

No, it has *always* been unsafe. What's new as of 10.13/14 is that macOS tries 
much harder at runtime to detect such cases and more predictably cause an error 
rather than letter than let the process run on and possibly fail 
nondeterministically. 

> Do only a few Python module use the Objective-C runtime? Or is it basically 
> "everything"?

I don't think we should try to second-guess this.  We now recognize that using 
fork like this on macOS has always been dangerous.  For some programs it will 
be fine, for others it won't.  People have had many macOS and Python releases 
to deal with this; if it works for their application, we shouldn't be changing 
the default for them.  But let's make it easier for new users to do the right 
thing - first by documenting the pitfall, than, in 3.8, changing the default.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

> To be clear, what is unsafe on macOS (as of 10.13, but even more so on 10.14) 
> is calling into the Objective-C runtime between fork and exec.  The problem 
> for Python is that it’s way too easy to do that implicitly, thus causing the 
> macOS to abort the subprocess in surprising ways.

Do only a few Python module use the Objective-C runtime? Or is it basically 
"everything"?

If it's just a few, would it be possible to emit a warning or even an exception 
if called in a child process after fork?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Documentation of the issue for sure!

A RuntimeWarning is a maybe if it can be made conditional on the OS version 
(10.13 and higher?)... when the "fork" method would be used.  BUT be very 
cautious about warnings as they tend to crop up in front of users rather than 
developers when it isn't their fault.  So if there is any flaw in the logic of 
when to show it in a situation where the application isn't about to crash or 
lockup, it'll show up when it shouldn't and cause somebody some grief to deal 
with via new code to ignore that warning.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On May 28, 2019, at 16:43, Ned Deily  wrote:

> Thanks, I was just going to add that I would accept a doc change for 3.7.  
> But the wording should be a little clearer that fork has *always* been unsafe 
> on macOS, i.e. this is not a new issue for 3.7 which is one of the reasons it 
> should not be backported.

To be clear, what is unsafe on macOS (as of 10.13, but even more so on 10.14) 
is calling into the Objective-C runtime between fork and exec.  The problem for 
Python is that it’s way too easy to do that implicitly, thus causing the macOS 
to abort the subprocess in surprising ways.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:

> Can we at least document that the default start method (fork) is now unsafe 
> on macOS?

Thanks, I was just going to add that I would accept a doc change for 3.7.  But 
the wording should be a little clearer that fork has *always* been unsafe on 
macOS, i.e. this is not a new issue for 3.7 which is one of the reasons it 
should not be backported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

Ned Deily:
> GPS beat me to it: this definitely should not be backported to either 3.7 or 
> 2.7 since it is a major user behavior change.

Hum ok, but test_multiprocessing_fork is now always skipped on macOS in Python 
3.7.

Can we at least document that the default start method (fork) is now unsafe on 
macOS? Maybe also explain how to change the default start method on macOS.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:

GPS beat me to it: this definitely should not be backported to either 3.7 or 
2.7 since it is a major user behavior change.

--
versions:  -Python 2.7, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Gregory P. Smith]

Gregory P. Smith  added the comment:

FWIW I am in favor of "spawn" being the default on _all_ platforms in 3.8.  The 
safest option being the default just seems like the right thing to do.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Gregory P. Smith]

Gregory P. Smith  added the comment:

There is a multiprocessing spawn method on Python 2.7.  And we should never do 
such a crazy huge feature backport.

IMNSHO - We should not change the default in 3.7 either.  That is a notable 
behavior change.  That decision is entirely up to Ned (RM).  If people are 
using an OS that changes behavior out from underneath their application between 
minor OS version upgrades, they should take that up with the OS vendor.  We 
cannot workaround this fundamental problem.

Applications and libraries will still need to support a slew of Python versions 
and OS versions, so they're all going to need to be modified to explicitly 
request the "spawn" method from multiprocessing on macOS anyways.  So there 
seems little point in changing the default within a patch release 3.7.4.

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On May 27, 2019, at 16:28, STINNER Victor  wrote:

> I don't see a clear consensus to switch to spawn on *all* platforms, so I 
> wrote PR 13603 which is the minimum fix: switch to spawn by default, but only 
> on macOS.

Fair enough.  Let’s fix what we have consensus on and go from there.  Thanks 
for working on this!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

Ok, I pushed a minimum fix to make sure that it lands in Python 3.8 beta1. 
Please check if the documentation if properly updated :-)

We will still have time after beta1 to revisit the question of making spawn the 
default on more/all platforms.

I wrote PR 13626 to change also Python 3.7. Once this one will be merged, I 
will also write a PR for Python 2.7.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13525
pull_request: https://github.com/python/cpython/pull/13626

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 17a5588740b3d126d546ad1a13bdac4e028e6d50 by Victor Stinner in 
branch 'master':
bpo-33725: multiprocessing uses spawn by default on macOS (GH-13603)
https://github.com/python/cpython/commit/17a5588740b3d126d546ad1a13bdac4e028e6d50


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

As soon as the "fork" start method "just works" on Linux, I'm not sure why you 
want to use "spawn" by default on Linux.

I don't buy the "consistency" argument here. I'm not surprised that such "low 
level" module like multiprocessing behaves differently depending on the 
platform to get the "native" and most efficient way to spawn subprocesses.

In general, fork is bad, but it's also convenient and people rely on it to 
prepare data in a main process and then "duplicate" the process to inherit 
cooked data.

If someone wants to more away from fork by default, I would suggest to *first* 
enhance multiprocessing documentation to list issues caused by fork(), and 
maybe also described when fork is safe (never doesn't sound like a good answer, 
since so far, I'm not aware of tons of bug reports on Linux caused by fork :-)).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

I don't see a clear consensus to switch to spawn on *all* platforms, so I wrote 
PR 13603 which is the minimum fix: switch to spawn by default, but only on 
macOS.

If this PR is merged, I understand that it should be applied to 2.7 and 3.7 
branches as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13509
pull_request: https://github.com/python/cpython/pull/13603

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Tim Smith]

Change by Tim Smith :


--
nosy: +tdsmith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Change by Ned Deily :


--
pull_requests:  -13382

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Matthias Bussonnier]

Change by Matthias Bussonnier :


--
pull_requests: +13382

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On May 10, 2019, at 04:07, Josh Rosenberg  wrote:
> 
> I've seen far too many cases where Python code targeting Linux intentionally 
> uses the COW benefits of fork for multiprocessing to think it would be a good 
> idea to change the default start method there without *some* sort of 
> deprecation period.

We need to resolve this for 3.8, and given that I think we have clear consensus 
to change the default on macOS to spawn to avoid the crashes, let’s do that.  
We’ll need to update the documentation.

Then if we don’t have consensus to change the default on Linux, let’s issue a 
DeprecationWarning for the default ‘fork’ method in 3.8 and change it to 
‘spawn' in 3.9.

Do we want to issue a warning on set_start_method(‘fork’) on macOS, given that 
it’s unlikely to be safe?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Josh Rosenberg]

Josh Rosenberg  added the comment:

I've seen far too many cases where Python code targeting Linux intentionally 
uses the COW benefits of fork for multiprocessing to think it would be a good 
idea to change the default start method there without *some* sort of 
deprecation period.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

I have no preference for the default on Linux, but please fix the default at 
least on macOS before Python 3.8 beta 1 ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On May 5, 2019, at 11:44, Davin Potts  wrote:
> 
> Victor raises an important question:  should the *default* start behavior be 
> made consistent across platforms?

Yes, I think it should.  The pros of consistency and correctness outweigh the 
breaking change IMHO.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Davin Potts]

Davin Potts  added the comment:

Victor raises an important question:  should the *default* start behavior be 
made consistent across platforms?  Assuming we change it on MacOS, the default 
start behavior on Windows and MacOS will be spawn but the default start 
behavior on Linux and FreeBSD (among others) will be fork.

Reasons to consider such a breaking change:
* This inconsistency in default start behavior on different platforms (Windows 
versus not) has historically been a significant source of confusion for many, 
many users.
* These days, the majority of users are not already familiar with the rule 
"fork-before-creating-threads" and so are surprised and confused when they fork 
a process that already has spun up multiple threads and bad things happen.
* We are changing the default on one platform (MacOS), which should prompt us 
to consider how are defaults are set elsewhere.

Reasons to reject such a breaking change:
* Though changing the default does not break everyone's code everywhere, it 
will require changes to any code that depends upon the default start method AND 
depends upon data/functions/stuff from the parent to also be present in the 
forked child process.

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Davin Potts]

Davin Potts  added the comment:

I believe we must change the default behavior on MacOS to use spawn instead of 
fork.  Encouraging people to use fork by default on MacOS is encouraging them 
to create something that effectively will not work.  Keeping fork as the 
default behavior when we have already turned off all of the tests of fork 
behavior on MacOS also makes no sense.  Existing Python code that depends upon 
the default behavior (fork) on MacOS has already been broken -- if we make this 
change, we are arguably not breaking anyone's working code.

Users can and will still be able to specify the start mechanism on MacOS, 
including fork.  This empowers users to continue to handle even the most 
esoteric use cases without loss of functionality from multiprocessing.  Though 
admittedly, without an ability to test the behavior of fork, this will need to 
be marked as deprecated.

I will supply a patch making this change and updating the docs shortly after 
PyCon.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

> And that change should be made for every platform so it will affect a much 
> broader group of users than are affected by *this* issue.

Do you mean to stop using fork on Linux as well? Using fork is just fine on 
Linux, no?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Łukasz Langa]

Łukasz Langa  added the comment:

It's trivial but not safe in the sense that code that previously depended on 
some global state setup done in the master process right before fork will stop 
working. If this code is a library that is not in your control, you might not 
be able to "just revert" to fork mode easily.

And that change should be made for every platform so it will affect a much 
broader group of users than are affected by *this* issue.

Another thing is that some application packagers (at least Facebook's XAR but 
probably many others) don't work with "start" by default.

So don't take this lightly, just as Davin is saying. That being said, it's 
probably wise to change the default to "start" which is a better method. And if 
we *are* changing, doing it sooner rather than later makes the most sense.

--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

Changing the default is trivial and safe. Let's do that. Additional doc 
wouldn't hurt.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Davin Potts]

Davin Potts  added the comment:

As best as I can see, there is no magic bullet to help mitigate this.

At a minimum, I am convinced we need to update the documentation to describe 
this behavior on MacOS and recommend alternatives.

I continue to give serious thought to the idea of changing the default start 
method on MacOS from fork to spawn.  This would be a breaking change though one 
could argue MacOS has already undergone a breaking change.  Is such a change 
warranted?

The alternative (which does not seem all that appealing) is that we start 
encouraging everyone to first consider the start method before attempting to 
use multiprocessing even for their first time.  Providing sensible defaults is 
to be preferred, but changing the default to reflect a non-trivial change in 
the underlying platform is still not to be taken lightly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:

What's the status of this issue?  As far as I know, we haven't come up with any 
workaround (see msg331011 above) other than disabling our tests.  If we don't 
have anything better, at the very least we should add a warning to the 
documentation to avoid fork mode on macOS, no?

--
priority: normal -> critical
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Change by Ned Deily :


--
pull_requests:  -10551

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Change by Ned Deily :


--
pull_requests:  -10552

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Tal Einat]

Change by Tal Einat :


--
pull_requests: +10551, 10552

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Tal Einat]

Change by Tal Einat :


--
pull_requests: +10551

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On Dec 12, 2018, at 17:59, Ned Deily  wrote:
> 
> Ned Deily  added the comment:
> 
>> Would it be safe to run the multiprocessing tests on recent macOS with the 
>> OBJC_DISABLE_INITIALIZE_FORK_SAFETY environment variable set?
> 
> See Ronald's reply above in msg331101. I believe his point is that there is 
> nothing you can do to make this safe. And it's not a new problem with 10.14 
> or 10.13. What is new is that Apple is trying to more forcefully make you 
> aware of the danger by causing the runtime to try to catch and crash these 
> cases earlier rather than permit them to perhaps silently cause failures 
> later.

In my experiments at least, setting the env var *does* prevent the crash, but 
it doesn’t avoid the undefined semantics (i.e. what happens when the ObjC 
runtime is called at that point?) and I fully expect that Apple will remove 
that bandaid at some point.

The other key thing is that I don’t believe you can set the env var *in 
process* and have it take effect after the fork.  It must be set before the 
parent process starts.  So that probably makes it less useful for the 
multiprocessing tests by itself.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:

> Would it be safe to run the multiprocessing tests on recent macOS with the 
> OBJC_DISABLE_INITIALIZE_FORK_SAFETY environment variable set?

See Ronald's reply above in msg331101. I believe his point is that there is 
nothing you can do to make this safe. And it's not a new problem with 10.14 or 
10.13. What is new is that Apple is trying to more forcefully make you aware of 
the danger by causing the runtime to try to catch and crash these cases earlier 
rather than permit them to perhaps silently cause failures later.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

Would it be safe to run the multiprocessing tests on recent macOS with the 
OBJC_DISABLE_INITIALIZE_FORK_SAFETY environment variable set?

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I think it make sense to disable this test; the only possible modification 
would be to only disable it for macOS <= 10.13.  AFAIK, that's the first 
version where core dumps were possible.  (Aside: I also saw these core dumps 
for a long time on 10.13 and never associated it with fork-without-exec in 
Python code.  Clearly, Apple has not done a good enough job of advertising this 
change.)

I think it is useful to help users on macOS avoid these problematic idioms, via 
documentation and defaults.  I think there's no way to predict when the core 
dumps will happen.  With internal cases, I've seen repeated invocations of the 
same code only core dump on the first run of the process, and not subsequent 
ones, for reasons I do not understand.  There seems to be a lot of mystery 
here, and without some explicit help from Apple, we're just doing our best to 
guess.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Davin Potts]

Davin Potts  added the comment:

@ned.deily: Apologies, I misread what you wrote -- I would like to see the 
random segfaults that you were seeing on Mojave if you can still point me to a 
few.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Davin Potts]

Davin Potts  added the comment:

Do we really need to disable the running of test_multiprocessing_fork entirely 
on MacOS?

My understanding so far is that not *all* of the system libraries on the mac 
are spinning up threads and so we should expect that there are situations where 
fork alone may be permissible, but of course we don't yet know what those are.  
Pragmatically speaking, I have not yet seen a report of 
test_multiprocessing_fork tests triggering this problem but I would like to 
see/hear that when it is observed (that's my pitch for leaving the tests 
enabled).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:

Since it looks like multiprocessing_fork is not going to be fixable for macOS, 
the main issue remaining is how to help users avoid this trap (literally).  
Should we add a check and issues a warning or error at run time?  Or is a doc 
change sufficient?

In the meantime, I've merged changes to disable running 
test_multiprocessing_fork which will sometimes (but not always) segfault on 
10.14 Mojave.  I should apologize to Barry and others who have run into this.  
I did notice the occasional segfault when testing with Mojave just prior to its 
release but it wasn't always reproducible and I didn't follow up on it.  Now 
that the change in 10.14 behavior makes this existing problem with fork no exec 
more obvious, it's clear that the test segfaults are another manifestation of 
this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[miss-islington]

miss-islington  added the comment:


New changeset df5d884defc8f1a94013ff9beb493f1428bd55b5 by Miss Islington (bot) 
in branch '3.6':
bpo-33725: skip test_multiprocessing_fork on macOS (GH-11043)
https://github.com/python/cpython/commit/df5d884defc8f1a94013ff9beb493f1428bd55b5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[miss-islington]

miss-islington  added the comment:


New changeset d4bcf13e06d33b8ec66a68db20df34a029e66882 by Miss Islington (bot) 
in branch '3.7':
bpo-33725: skip test_multiprocessing_fork on macOS (GH-11043)
https://github.com/python/cpython/commit/d4bcf13e06d33b8ec66a68db20df34a029e66882


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[miss-islington]

Change by miss-islington :


--
pull_requests: +10280

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:


New changeset ac218bc5dbfabbd61c76ce8a17de088611e21981 by Ned Deily in branch 
'master':
bpo-33725: skip test_multiprocessing_fork on macOS (GH-11043)
https://github.com/python/cpython/commit/ac218bc5dbfabbd61c76ce8a17de088611e21981


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[miss-islington]

Change by miss-islington :


--
pull_requests: +10281

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Change by Ned Deily :


--
keywords: +patch
pull_requests: +10279
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ronald Oussoren]

Ronald Oussoren  added the comment:

AFAIK there is nothing you can do between after calling fork(2) to 
"reinitialise" the ObjC runtime. And I don't think that's the issue anyway: I 
suspect that the actual problem is that Apple's system frameworks use 
multithreading (in particular libdispatch) and don't have code to ensure a sane 
state after calling fork. 

In Python 3 there is another workaround to avoid problems using 
multiprocessing: use multiprocessing.set_start_method() to switch away from the 
"fork" startup handler to "spawn" or "forkserver" (the latter only when calling 
set_start_method before calling any code that might call into Apple system 
frameworks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I have a reliable way to call *something* in the pthread_atfork prepare 
handler, but I honestly don't know what to call to prevent the crash.

In the Ruby thread, it seemed to say that you could just dlopen 
/System/Library/Frameworks/Foundation.framework/Foundation but that does not 
work for me.  Neither does also loading the CoreFoundation and 
SystemConfiguration frameworks.

If anybody has something that will reliably initialize the runtime, I can post 
my approach (there are a few subtleties).  Short of that, I think there's 
nothing that can be done except ensure that exec is called right after fork.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On Nov 14, 2018, at 10:11, Davin Potts  wrote:
> 
> 
> Davin Potts  added the comment:
> 
> Barry's effort as well as comments in other links seem to all suggest that 
> OBJC_DISABLE_INITIALIZE_FORK_SAFETY is not comprehensive in its ability to 
> make other threads "safe" before forking.

Right.  Setting the env var will definitely not make it thread safe.  My 
understanding (please correct me if I’m wrong!) isn’t that this env var makes 
it safe, just that it prevents the ObjC runtime from core dumping.  So it’s 
still up to the developer to know whether threads are involved or not.  In our 
cases, these are single threaded applications.  I’ve read elsewhere that ObjC 
doesn’t care if threads have actually been spun up or not.

> "Objective-C classes defined by the OS frameworks remain fork-unsafe" (from 
> @kapilt's first link) suggests we furthermore remain at risk using certain 
> MacOS system libraries prior to any call to fork.

Actually, it’s unsafe to call anything between fork and exec.  Note that this 
doesn’t just affect Python; this is a pretty common idiom in other scripting 
languages too, from what I can tell.  It’s certainly very common in Python.

Note too that urllib.request.getproxies() will end up calling into the ObjC 
runtime via _scproxy, so you can’t even use requests after a fork but before 
exec.

What I am still experimenting with is to see if I can define a pthread_atfork 
handler that will initialize the ObjC runtime before fork is actually called.  
I saw a Ruby approach like this, but it’s made more difficult in Python because 
pthread_atfork isn’t exposed to Python.  I’m trying to see if I can implement 
it in ctypes, before I write an extension.

> "To guarantee that forking is safe, the application must not be running any 
> threads at the point of fork" (from @kapilt's second link) is an old truth 
> that we continue to fight with even when we know very well that it's the 
> truth.

True, but do realize this problem affects you even in single threaded 
applications.

> For newly developed code, we have the alternative to employ spawn instead of 
> fork to avoid these problems in Python, C, Ruby, etc.  For existing legacy 
> code that employed fork and now surprises us by failing-fast on MacOS 10.13 
> and 10.14, it seems we are forced to face a technical debt incurred back when 
> the choice was first made to spin up threads and afterwards to use fork.

It’s tech debt you incur even if you don’t spin up threads.  Just fork and do 
some work in the child before calling exec.  If that work enters the ObjC 
runtime (as in the getproxies example), your child will coredump,

> If we didn't already have an "obvious" (zen of Python) way to avoid such 
> problems with spawn versus fork, I would feel this was something to solve in 
> Python.  As to helping the poor unfortunate souls who must fight the good 
> fight with legacy code, I am not sure what to do to help though I would like 
> to be able to help.

*If* we can provide a hook to initialize the ObjC runtime in pthread_atfork, I 
think that’s something we could expose in Python.  Then we can say legacy code 
can just invoke that, and at least you will avoid the worst outcome.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Davin Potts]

Davin Potts  added the comment:

Given the original post mentioned 2.7.15, I wonder if it is feasible to fork 
near the beginning of execution, then maintain and pass around a 
multiprocessing.Pool to be used when needed instead of dynamically forking?  
Working with legacy code is almost always more interesting than you want it to 
be.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Ned Deily]

Ned Deily  added the comment:

_scproxy has been known to be problematic for some time, see for instance 
Issue31818.  That issue also gives a simple workaround: setting urllib's 
"no_proxy" environment variable to "*" will prevent the calls to the System 
Configuration framework.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Legacy code is easy to migrate as long as it uses Python 3.  Just call

  mp.set_start_method('forkserver')

at the top of your code and you're done.  Some use cases may fail (if sharing 
non-picklable types), but they're probably not very common.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Davin Potts]

Davin Potts  added the comment:

Barry's effort as well as comments in other links seem to all suggest that 
OBJC_DISABLE_INITIALIZE_FORK_SAFETY is not comprehensive in its ability to make 
other threads "safe" before forking.

"Objective-C classes defined by the OS frameworks remain fork-unsafe" (from 
@kapilt's first link) suggests we furthermore remain at risk using certain 
MacOS system libraries prior to any call to fork.

"To guarantee that forking is safe, the application must not be running any 
threads at the point of fork" (from @kapilt's second link) is an old truth that 
we continue to fight with even when we know very well that it's the truth.

For newly developed code, we have the alternative to employ spawn instead of 
fork to avoid these problems in Python, C, Ruby, etc.  For existing legacy code 
that employed fork and now surprises us by failing-fast on MacOS 10.13 and 
10.14, it seems we are forced to face a technical debt incurred back when the 
choice was first made to spin up threads and afterwards to use fork.

If we didn't already have an "obvious" (zen of Python) way to avoid such 
problems with spawn versus fork, I would feel this was something to solve in 
Python.  As to helping the poor unfortunate souls who must fight the good fight 
with legacy code, I am not sure what to do to help though I would like to be 
able to help.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

FWIW, I suspect that setting the environment variable only helps if it's done 
before the process starts.  You cannot set it before the fork and have it 
affect the child.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

A few other things I don't understand:

* Why does setting OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES only seem to work 
when it's set in the shell before the parent process executes?  AFAICT, it does 
*not* work if you set that in os.environ in the parent process before the 
os.fork().

* Why does it only crash on the first invocation of our app?  Does getproxies() 
cache the results somehow?  There's too much internal application code in the 
way to know if we're doing something that prevents getproxies() from getting 
called in subsequent calls.

* I can't seem to produce a smaller test case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Hoo boy.  I'm not sure I have the full picture, but things are starting to come 
into focus.  After much debugging, I've narrowed down at least one crash to 
urllib.request.getproxies().  On macOS (darwin), this ends up calling 
_scproxy.get_proxies() which calls into the SystemConfiguration framework.  
I'll bet dollars to donuts that that calls into the ObjC runtime.  Thus it is 
unsafe to call between fork and exec.  This certainly seems to be the case even 
if the environment variable is set.

The problem is that I think requests.post() probably also ends up in here 
somehow (still untraced), because by removing our call to 
urllib.requests.getproxies(), we just crash later on when requests.post() is 
called.

I don't know what, if anything can be done in Python, except perhaps to 
document that anything that calls into the ObjC runtime between fork and exec 
can potentially crash the subprocess.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
title: Pytho crashes on macOS after fork with no exec -> Python crashes on 
macOS after fork with no exec

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com