[issue23213] subprocess communicate() hangs when stderr isn't closed

[Thomas D.]

Thomas D. added the comment:

Yep, I agree with you. This can be closed.

Thanks to your clarification the problem could be resolved upstream.

The udev problem was resolved in https://github.com/systemd/systemd/issues/190 
and the PHP problem has a pending PR https://github.com/php/php-src/pull/1432

--
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23213] subprocess communicate() hangs when stderr isn't closed

[Martin Panter]

Martin Panter added the comment:

I suggest closing this. Depending on what the daemons are supposed to do, I 
think this is either a bug with “systemd” and PHP holding onto stderr when they 
become daemons, or a bug/misunderstanding with how Thomas was using the 
subprocess module and communicate().

--
resolution:  -> third party
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23213] subprocess communicate() hangs when stderr isn't closed

[Martin Panter]

Martin Panter added the comment:

I suspect this is not a bug but a misunderstanding of how communiate(), pipes, 
daemon processes, etc, work. If communicate() didn’t wait for stderr to be 
closed, then how would it know it had read all the data that was written into 
the pipe?

I don’t have that version of “systemd”, but I guess your daemon leaves its 
output pipes open and continues to run in the background, so communicate() is 
waiting in case the daemon writes anything to the pipes. This would demonstrate 
the same situation using a Python subprocess:

import subprocess
import sys
script = import os, time
if not os.fork():  # Child process
time.sleep(30)
print(Finally!)

args = (sys.executable, -c, script)
proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.communicate()
# Hangs for 30 s and then returns (b'Finally!\n', b'')

If you want communicate() to return as soon as a daemon forks into the 
background, don’t read from the daemon’s output pipes. If you want to read data 
that the foreground process writes and ignore any data that the background 
process might write to the same pipes, that’s asking for race conditions.

--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23213
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23213] subprocess communicate() hangs when stderr isn't closed

[Thomas D.]

New submission from Thomas D.:

Hi,

to demonstrate the problem you need =systemd-217:

# python3.4
Python 3.4.2 (default, Oct 12 2014, 20:09:43)
[GCC 4.8.3] on linux
Type help, copyright, credits or license for more information.
 import subprocess
 sp = subprocess.Popen([/sbin/udevd, --daemon], stdout=subprocess.PIPE, 
 stderr=subprocess.PIPE)
 out, err = sp.communicate()

[hangs]

ps will show

root   9619  0.0  0.1  23340  5404 pts/5Ss   Jan09   0:00  \_ -bash
root  13291  0.0  0.2  45352  9784 pts/5S+   00:34   0:00  \_ 
python3.4
root  13311  0.0  0.0  0 0 pts/5Z+   00:34   0:00  \_ 
[udevd] defunct

Calling /sbin/udevd --daemon from the shell works fine.

 errorlog = open(/tmp/stderr.log, wb)
 sp = subprocess.Popen([/sbin/udevd, --daemon], stdout=subprocess.PIPE, 
 stderr=errorlog)

works, too.

The problem first appeared in systemd-217. I bisected systemd's source code and 
the commit since when Python's subprocess module is unable to start udevd is

https://github.com/systemd/systemd/commit/5c67cf2774a8b964f4d7cd92a4c447da81ac6087


This is not a systemd/udev only problem. The problem was first seen with the 
php-fpm daemon from PHP (but only when using error_log = syslog).

Please see the original bug report at 
https://github.com/saltstack/salt/issues/14957 for more details.

Because Salt is still at Python 2.7, the problem can be reproduced with Python 
2.7, too.


Is it a bug in subprocess? In systemd/PHP? Are we (salt) using subprocess the 
wrong way?

Thanks!


PS:
On your system, /sbin/udevd will be probably /lib/systemd/systemd-udevd

Not sure if this is related to http://bugs.python.org/issue12786 in some ways.

--
components: Library (Lib)
messages: 233788
nosy: whissi
priority: normal
severity: normal
status: open
title: subprocess communicate() hangs when stderr isn't closed
type: behavior
versions: Python 2.7, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23213
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com