[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[STINNER Victor]

STINNER Victor  added the comment:

> I'm reducing the severity from release blocker to high and keep the ticket in 
> pending to give Eric a change to review the commits.

Python 3.8.0 is released with the fix. It's now time to close the issue.

--
stage: commit review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Christian Heimes]

Christian Heimes  added the comment:

https://github.com/freeipa/freeipa/pull/3769 should address the issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Christian Heimes]

Christian Heimes  added the comment:

I'll address the issue in FreeIPA.

The ipautil.run() function is a helper around subprocess.Popen. The function 
always installs a preexec_fn in case it needs to change umask or drop 
priviliges. The WSGI server does not need these features.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Adam Williamson]

Adam Williamson  added the comment:

It's this function:

https://github.com/freeipa/freeipa/blob/master/ipalib/install/kinit.py#L66

The function `run` is imported from `ipapython.ipautil`, it's defined here:

https://github.com/freeipa/freeipa/blob/master/ipapython/ipautil.py#L391

all of this is being run inside a WSGI.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Gregory P. Smith]

Gregory P. Smith  added the comment:

preexec_fn is fundamentally unsupportable.

what code is using it, there should be a way not to rely on that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Adam Williamson]

Adam Williamson  added the comment:

Well, now our (Fedora QA's) automated testing of FreeIPA is showing what looks 
like a problem with preexec_fn (rather than fork) being disallowed:

https://bugzilla.redhat.com/show_bug.cgi?id=1759290

Login to the FreeIPA webUI is failing, and at the time it fails we see this 
error message on the server end:

[Mon Oct 07 09:22:19.521604 2019] [wsgi:error] [pid 32989:tid 139746234119936] 
[remote 10.0.2.102:56054] ipa: DEBUG: args=['/usr/bin/kinit', 'admin', '-c', 
'/run/ipa/ccaches/kinit_32989', '-E']
[Mon Oct 07 09:22:19.521996 2019] [wsgi:error] [pid 32989:tid 139746234119936] 
[remote 10.0.2.102:56054] ipa: DEBUG: Process execution failed
[Mon Oct 07 09:22:19.522189 2019] [wsgi:error] [pid 32989:tid 139746234119936] 
[remote 10.0.2.102:56054] ipa: INFO: 401 Unauthorized: preexec_fn not supported 
within subinterpreters

--
nosy: +adamwill
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Christian Heimes]

Christian Heimes  added the comment:

Thanks Victor and Gregory!

I'm reducing the severity from release blocker to high and keep the ticket in 
pending to give Eric a change to review the commits.

--
priority: release blocker -> high
resolution:  -> fixed
stage: patch review -> commit review
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[miss-islington]

miss-islington  added the comment:


New changeset 03c52f2f63a8abeb4afb75e9da46c7d6c0a8afd5 by Miss Islington (bot) 
in branch '3.8':
bpo-37951: Lift subprocess's fork() restriction (GH-15544)
https://github.com/python/cpython/commit/03c52f2f63a8abeb4afb75e9da46c7d6c0a8afd5


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[miss-islington]

Change by miss-islington :


--
pull_requests: +15229
pull_request: https://github.com/python/cpython/pull/15554

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Christian Heimes]

Christian Heimes  added the comment:


New changeset 98d90f745d35d5d07bffcb46788b50e05eea56c6 by Christian Heimes in 
branch 'master':
bpo-37951: Lift subprocess's fork() restriction (GH-15544)
https://github.com/python/cpython/commit/98d90f745d35d5d07bffcb46788b50e05eea56c6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Christian Heimes]

Christian Heimes  added the comment:

I have created a PR that implements Greg's proposal 
https://bugs.python.org/issue34651#msg325302

--
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Christian Heimes]

Change by Christian Heimes :


--
keywords: +patch
pull_requests: +15221
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/15544

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Gregory P. Smith]

Gregory P. Smith  added the comment:

FWIW, _posixsubprocess.fork_exec() should be safe to allow.

The only thing within it to disallow, if you're going to bother to check this 
at all, is any use of the legacy preexec_fn support.

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Łukasz Langa]

Łukasz Langa  added the comment:

Christian, you're right to treat this as Release Blocker. Let's have this 
fixed. Assigning Eric?

--
assignee:  -> eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Christian Heimes]

Christian Heimes  added the comment:

It's a bit more complicated. FreeIPA uses cryptography, which uses asn1crypto, 
which uses ctypes, which is broken in mod_wsgi due to bpo-34651. It's not just 
FreeIPA that is affected by the issue. Any application running in mod_wsgi is 
potentially affected and broken by bpo-34651.

1a) (modify FreeIPA) is not possible. IPA requires the additional features of 
the subprocess module.
1b) (modify ctypes) should be done in a separate ticket. I'm not sure why 
subprocess does not use posix_spawn() here. I guess it's the default value 
"close_fds=True"?
2) (avoid subinterpreters) would require a rewrite of mod_wsgi
3) (revert bpo-34651) is IMHO required for _posixsubprocess.fork_exec().

bpo-34651 is a backwards incompatible change that breaks existing applications 
that uses mod_wsgi. At least _posixsubprocess.fork_exec() should be reverted 
and the removal of fork() support should go through a proper deprecation cycle 
of two releases.

I'm bumping this up to release blocker and CC Łukasz.

--
nosy: +lukasz.langa
priority: critical -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[STINNER Victor]

STINNER Victor  added the comment:

subprocess still work in subinterpreters in Python 3.8 if posix_spawn() can be 
used, but posix_spawn() is only used under some conditions:
https://docs.python.org/dev/whatsnew/3.8.html#optimizations

"The subprocess module can now use the os.posix_spawn() function in some cases 
for better performance. Currently, it is only used on macOS and Linux (using 
glibc 2.24 or newer) if all these conditions are met:

* close_fds is false;
* preexec_fn, pass_fds, cwd and start_new_session parameters are not set;
* the executable path contains a directory."

--

It seems like FreeIPA uses ctypes and ctypes calls 
subprocess.Popen(['/sbin/ldconfig', '-p'], ...) to locale libcrypto.

I see different options:

* modify FreeIPA / ctypes to ensure that posix_spawn() can be used
* avoid subinterpreters to deploy FreeIPA
* revert the change to allow again fork in subprocesses: see bpo-34651 for the 
rationale why it was denied

I understand that FreeIPA is run as WSGI using mod_wsgi in Apache.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37951] Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

[Christian Heimes]

New submission from Christian Heimes :

BPO https://bugs.python.org/issue34651 disabled fork in subinterpreters. The 
patch also disabled fork() in _posixsubprocess.fork_exec(). This broke the 
ability to spawn subprocesses in mod_wsgi daemons, which use subinterpreters. 
Any attempt to spawn (fork + exec) a subprocess fails with "RuntimeError: fork 
not supported for subinterpreters":

...
  File "/usr/lib64/python3.8/subprocess.py", line 829, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.8/subprocess.py", line 1608, in _execute_child
self.pid = _posixsubprocess.fork_exec(
RuntimeError: fork not supported for subinterpreters

Also see https://bugzilla.redhat.com/show_bug.cgi?id=1745450

--
components: Extension Modules, Interpreter Core
keywords: 3.8regression
messages: 350511
nosy: christian.heimes, eric.snow, vstinner
priority: critical
severity: normal
status: open
title: Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon 
mode
versions: Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com