[issue33725] High Sierra hang when using multi-processing

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

issue35219 is where I've run into this problem.  I'm still trying to figure out 
all the details in my own case, but I can confirm that setting the environment 
variable does not always help.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
versions: +Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Ned Deily]

Ned Deily  added the comment:

{Note: this is not particularly relevant to the issue here.)

Ronald:
> In the long run the macOS installers should be build using the latest SDK 
> [...] That's something that would require some effort though to ensure that 
> the resulting binary still works on older versions of macOS

I agree that being able to build with the latest SDK would be nice but it's 
also true it would require effort on our part, both one-time and ongoing, at 
least for every new macOS SDK release and update to test with each older 
system.  It would also require that the third-party libraries we build for an 
installer also behave correctly.  And to make full use of it, third-party 
Python packages with extension modules would also need to behave correctly.  I 
see one of the primary use cases for the python.org macOS installers as being 
for Python app developers who want to provide apps that run on a range of macOS 
releases.  It seems to me that the safest and simplest way to guarantee that 
python.org macOS Pythons fulfill that need is to continue to always build them 
on the oldest supported system.  Yes, that means that users may miss out on a 
few features only supported on the more recent macOS releases but I think 
that's the right trade-off until we have the resources to truly investigate
  and decide to support weak linking from current systems.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Ronald Oussoren]

Ronald Oussoren  added the comment:

@Ned: In the long run the macOS installers should be build using the latest 
SDK, primarily to get full API coverage and access to all system APIs.

AFAIK building using the macOS 10.9 SDK still excludes a number of libSystem 
APIs that would be made available through the posix module when building with a 
newer SDK. 

That's something that would require some effort though to ensure that the 
resulting binary still works on older versions of macOS (basically similar to 
the work I've done in the post to weak link some other symbols in the posix 
module).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Ronald Oussoren]

Ronald Oussoren  added the comment:

Antoine, the issue is not necessarily related to POSIX compliance, AFAIK 
strictly POSIX compliant code should work just fine. The problem is in 
higher-level APIs (CoreFoundation, Foundation, AppKit, ...), and appears to be 
related to using multi-threading in those libraries without spending effort on 
pre/post fork handlers to ensure that new processes are in a sane state after 
fork().  In older macOS versions this could result in hard to debug issues, in 
newer versions APIs seem to guard against this by aborting when the detect that 
the pid changed.

Anyways... I agree that we shouldn't try to work around this in CPython, 
there's bound to more problems that are hidden with the proposed workaround.

---


 describes what the environment variable does, and this "just" changes behavior 
of the ObjC runtime, and doesn't make using macOS system frameworks after a 
fork saver.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I understand that Apple, with their limited resources, cannot spend expensive 
engineer manpower on improving POSIX support in macOS .

In any case, I'm unsure this bug can be fixed at the Python level.  If macOS 
APIs don't like fork(), they don't like fork(), point bar.  As Ronald says, on 
3.x you should use "forkserver" (for multiple reasons, not only this issue).  
On 2.7 you're stuck dealing with the issue by yourself.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Ned Deily]

Ned Deily  added the comment:

(As a side note, the macOS Pythons provided by python.org installers should not 
behave differently on macOS 10.13 High Sierra since none of them are built with 
a 10.13 SDK.)

--
nosy: +davin, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Ronald Oussoren]

Ronald Oussoren  added the comment:

A better solution is to avoid using fork mode for multiprocessing. The spawn 
and fork server modes should work fine. 

The underlying problem is that macOS system frameworks (basically anything 
higher level than libc) are not save wrt fork(2) and fixing that appears to 
have no priority at all at Apple.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Kapil Thangavelu]

New submission from Kapil Thangavelu :

This issue seems to be reported a few times on various githubs projects. I've 
also reproduced using a brew install of python 2.7.15. I haven't been able to 
reproduce with python 3.6. Note this requires a framework build of python.

Background on the underlying issue cause due to a change in high Sierra 
http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
A ruby perspective on the same issue exhibiting for some apps
https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/


The work around seems to be setting an environment variable 
OBJC_DISABLE_INITIALIZE_FORK_SAFETY prior to executing python.

Other reports

https://bugs.python.org/issue30837
https://github.com/ansible/ansible/issues/32499
https://github.com/imWildCat/scylla/issues/22
https://github.com/elastic/beats-tester/pull/73
https://github.com/jhaals/ansible-vault/issues/60

--
components: macOS
messages: 318352
nosy: kapilt, ned.deily, ronaldoussoren
priority: normal
severity: normal
status: open
title: High Sierra hang when using multi-processing
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com