[issue1731717] race condition in subprocess module

[Krishna Oza]

Krishna Oza added the comment:

Hi, Could anyone help here to identify in which Python release the bug is 
fixed. I am unable to deduce from the bug tracker interface in which release 
this issue is fixed.

Regards.

--
nosy: +Krishna Oza

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Vitaly]

Vitaly added the comment:

Is this issue fixed in python 2.7.10? I am experiencing strange race conditions 
in code that combines threads with multiprocessing pool, whereby a thread is 
spawned by each pool worker. Running on latest Mac OS X.

--
nosy: +vitaly

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[jan matejek]

jan matejek jmate...@suse.cz added the comment:

please check my logic here, but the patched code seems to throw away perfectly 
valid return codes:
in wait(), self._handle_exitstatus(sts) gets called unconditionally, and it 
resets self.returncode also unconditionally.
now, if a _cleanup() already did _internal_poll and set self.returncode that 
way, it is lost when wait() catches the ECHILD, in the one place where it 
actually matters, by setting sts=0 for the _handle_exitstatus call

IMHO it could be fixed by moving _handle_exitstatus to the try: section, and 
returning self.returncode or 0 or something like that

--
nosy: +matejcik

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Gregory P. Smith]

Gregory P. Smith g...@krypto.org added the comment:

r87233 fixes the OSError escaping from wait() issue when SIGCLD is set to be 
ignored.  (to appear in 3.2beta1; it is a candidate for backporting to 3.1 and 
2.7)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Gregory P. Smith]

Gregory P. Smith g...@krypto.org added the comment:

sorry, i meant 3.2beta2 above.
release27-maint: r87234 targeting 2.7.2
release31-maint: r87235 targeting 3.1.4

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

It seems the canonical spelling is SIGCHLD. SIGCLD doesn't exist everywhere and 
it produces test failures under OS X:

http://www.python.org/dev/buildbot/all/builders/AMD64%20Leopard%203.x
http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%203.x

FWIW, this is what POSIX says:

“Some implementations, including System V, have a signal named SIGCLD, which is 
similar to SIGCHLD in 4.2 BSD. POSIX.1 permits implementations to have a single 
signal with both names. POSIX.1 carefully specifies ways in which conforming 
applications can avoid the semantic differences between the two different 
implementations. The name SIGCHLD was chosen for POSIX.1 because most current 
application usages of it can remain unchanged in conforming applications. 
SIGCLD in System V has more cases of semantics that POSIX.1 does not specify, 
and thus applications using it are more likely to require changes in addition 
to the name change.”

http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xsh_chap02.html

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[James Lamanna]

James Lamanna jlama...@gmail.com added the comment:

stubbing out subprocess._cleanup does not work around the problem from this 
example on 2.6.5:

import subprocess, signal
subprocess._cleanup = lambda: None

signal.signal(signal.SIGCLD, signal.SIG_IGN)
subprocess.Popen(['echo','foo']).wait()

ja...@hyla:~$ python tt.py
foo
Traceback (most recent call last):
  File tt.py, line 5, in module
subprocess.Popen(['echo','foo']).wait()
  File /usr/lib/python2.6/subprocess.py, line 1170, in wait
pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File /usr/lib/python2.6/subprocess.py, line 465, in _eintr_retry_call
return func(*args)
OSError: [Errno 10] No child processes

This bug still prevents subprocess from being used inside of a daemon where 
SIGCLD is being caught to reap zombie processes.

--
nosy: +jlamanna

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Gregory P. Smith]

Gregory P. Smith g...@krypto.org added the comment:

A workaround for those still having problems with this:

 stub out subprocess._cleanup with a no-op method.

It it only useful if your app is ever using subprocess and forgetting to call 
wait() on Popen objects before they are deleted.  If you are, you can keep a 
reference to the old _cleanup() method and periodically call it on your own.

http://bugs.python.org/issue1236 effectively demonstrates this.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Jonas H.]

Changes by Jonas H. jo...@lophus.org:


--
nosy: +jonash

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Guido van Rossum]

Changes by Guido van Rossum gu...@python.org:


--
nosy:  -gvanrossum

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[red]

red stefano.bon...@gmail.com added the comment:

I'm using an old Plone/Zope Product, PHParser, that uses the popen2 call ... 
same problem for me.
Is there a thread-safe alternative to execute subprocesses in threads? I need a 
patch!!! thanks in advance!!!

--
nosy: +shaphiro

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Collin Winter]

Changes by Collin Winter coll...@gmail.com:


--
nosy: +collinwinter, jyasskin

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Giampaolo Rodola']

Changes by Giampaolo Rodola' g.rod...@gmail.com:


--
nosy: +giampaolo.rodola

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Stefan]

Stefan s.ant...@telekom.de added the comment:

I have exactly the same problem. Is there a thread-safe alternative to execute 
subprocesses in threads?

--
nosy: +santoni

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Florent Xicluna]

Florent Xicluna florent.xicl...@gmail.com added the comment:

Previous buildbot failures were in test_multiprocessing:
http://bugs.python.org/issue1731717#msg100430

Now it should be fixed:
 - r78777, r78787, r78790 on 2.x
 - r78798 on 3.x

--
resolution:  - fixed
stage: test needed - committed/rejected
status: open - pending

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Yonas yona...@gmail.com added the comment:

Florent,

Have you tested any of the sample test programs mentioned in this bug report? 
For example, the one by Joel Martin (kanaka).

I'd suggest to test those first before marking this issue as fixed.

- Yonas

--
status: pending - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Florent Xicluna]

Florent Xicluna florent.xicl...@gmail.com added the comment:

 import subprocess, signal
 signal.signal(signal.SIGCLD, signal.SIG_IGN)
0
 subprocess.Popen(['echo','foo']).wait()
foo
Traceback (most recent call last):
  File stdin, line 1, in module
  File ./Lib/subprocess.py, line 1229, in wait
pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File ./Lib/subprocess.py, line 482, in _eintr_retry_call
return func(*args)
OSError: [Errno 10] No child processes


You're right, I fixed something giving the same ECHILD error in 
multiprocessing. But it was not related.

--
resolution: fixed - 
stage: committed/rejected - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Florent Xicluna]

Changes by Florent Xicluna florent.xicl...@gmail.com:


--
keywords:  -buildbot

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Yonas yona...@gmail.com added the comment:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/9a853d0308c8e55a


I'm also glad to see a test case that causes exactly the same error with or 
without the presence of a ‘daemon.DaemonContext’.

Further research shows that handling of ‘SIGCLD’ (or ‘SIGCLD’) is fairly
OS-specific, with “ignore it” or “handle it specifically” being correct
on different systems. I think Python's default handling of this signal
is already good (modulo bug #1731717 to be addressed in ‘subprocess’).

So I will apply a change similar to Joel Martin's suggestion, to default
to avoid touching the ‘SIGCLD’ signal at all, and with extra notes in
the documentation that anyone using child processes needs to be wary of
signal handling.

This causes the above test case to succeed; the output file contains:: 
=
Child process via os.system.
Child process via 'subprocess.Popen'.
Parent daemon process.
Parent daemon process done.
=

  -- By Ben Finney

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Yonas yona...@gmail.com added the comment:

Ben Finney's comment suggests to me that this bug is being ignored. Am I wrong?

with extra notes in the documentation that anyone using child processes needs 
to be wary of signal handling.

Why should they be wary? We should just fix this bug.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Yonas yona...@gmail.com added the comment:

By the way, in three months from today, this bug will be 3 years old.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Gregory P. Smith]

Gregory P. Smith g...@krypto.org added the comment:

I really don't care about the age of a bug.  This bug is young.  I've fixed 
many bugs over twice its age in the past.

Regardless, I've got some serious subprocess internal refactoring changes 
coming in the very near future to explicitly deal with thread safety issues.  
I'm adding this one to my list of things to tackle.

--
assignee: astrand - gregory.p.smith

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Yonas yona...@gmail.com added the comment:

Gregory,

Awesome! Approx. how long until we hear back from you in this report?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Florent Xicluna]

Florent Xicluna florent.xicl...@gmail.com added the comment:

Sometimes IA64 Ubuntu bot fails on this one.
http://www.python.org/dev/buildbot/all/builders/ia64%20Ubuntu%20trunk/builds/571

--
keywords: +buildbot

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Joel Martin]

Joel Martin pythonb...@martintribe.org added the comment:

I can reproduce the problem (or at least get the same symptom) by doing
this (in 2.4.6, 2.5.4 and 2.6.2):

import subprocess, signal
signal.signal(signal.SIGCLD, signal.SIG_IGN)
subprocess.Popen(['echo','foo']).wait()

The echo command completes, but the subprocess command throws the no
child exception. It seems like the subprocess command should either:

- detect that SIGCLD is not set correctly and throw a more informative
exception before running the command.
- override SIGCLD during the running of the sub-command. Not sure what
the UNIX correctness implications of this are.
- or allow the child to zombie without throwing an exception. The fact
that the programmer has overridden SIGCLD sort of implies that reaping
of zombie children has been switched off.

I don't have good enough understanding of the underlying implementation
to know if this is a reproducer as requested or if this should be a new
bug. Please advise and I will file a new bug if requested.

This is a follow-up to trying to resolve this problem in the PEP 3143
python-daemon module. See this thread:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/9a853d0308c8e55a

--
nosy: +kanaka
status: pending - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

I have confirmed that none of the original test cases referenced here or
in the referenced issues fails on python 2.5.4, 2.6.2, trunk, or py3k on
linux.

The test case attached to this ticket I cannot test as I don't
understand how one would run it (and exim appears to be involved).

Given the discussion (specifically msg32219) I think this ticket should
be closed as fixed; unless, Yonas, you can provide a reproducible test
case that does not involve third party software, or someone else can
reproduce your exim case.

If someone wants to propose a patch to refactor subprocess, IMO that
should be opened as a new feature request ticket.

--
nosy: +r.david.murray
resolution:  - fixed
status: open - pending

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Yonas yona...@gmail.com added the comment:

To test with exim4 is easy. To reproduce on Ubuntu Jaunty:

1. apt-get install exim4-daemon-heavy

2. echo local_scan = /usr/lib/exim4/local_scan/libpyexim.so 
/etc/exim4/conf.d/main/15_py-exim_plugin_path

3. cd /usr/lib/exim4/local_scan

4. Compile mylib, output will be libpyexim.so:

gcc `python2.6-config --cflags` -c -fPIC mylib.c -o mylib.o 
gcc -Xlinker -export-dynamic -lpython2.6 -lnsl -lpthread -ldl -lutil -lm
-lz -shared -Wl,-soname,libpyexim.so -o libpyexim.so  mylib.o

5. Restart server:
/etc/init.d/exim4 restart

6. Send some mail:
cat mail.txt | sendmail -t

(contents of mail.txt):
Content-type: text/plain
To: your_usern...@localhost
From: foo...@example.com
Subject: test

Hello world.

--
status: pending - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Yonas yona...@gmail.com added the comment:

Also, copy exim_local_scan2.py to /usr/lib/python2.6/

--
Added file: http://bugs.python.org/file14134/exim_local_scan2.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Yonas yona...@gmail.com added the comment:

I'm assuming that exim4 is reading config files from /etc/exim4/conf.d/*.

To make sure, you can enforce split file mode by running `sudo
dpkg-reconfigure exim4-config`. It should be one of the last questions
present to you.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

I'm afraid I'm not going to be installing exim in order to test this. 
Perhaps someone else will be interested enough to do so, perhaps not :)

Copying files into /usr/lib/pythonx.x is a very odd thing to do, by the
way (though it should have no impact on the bug).

--
priority: critical - normal
status: open - pending

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Yonas yona...@gmail.com added the comment:

I always get a subprocess error when using embedded python 2.6.2:

File /usr/lib/python2.6/subprocess.py, line 1123, in wait: pid,
sts = os.waitpid(self.pid, 0): OSError: [Errno 10] No child processes

Example library and main program are attached.

--
nosy: +yonas
Added file: http://bugs.python.org/file14086/mylib.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Yonas]

Changes by Yonas yona...@gmail.com:


Added file: http://bugs.python.org/file14087/main.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[djc]

Changes by djc dirk...@ochtman.nl:


--
nosy: +djc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Gabriel]

Gabriel m...@evotex.ch added the comment:

I had this happen to me today, I used Popen and os.waitpid(p.pid, 0) and
I got an OSError: [Errno 10] No child processes. I haven't tried
adding a sleep, since it's unreliable. I'm using python 2.5.2, Windows
XP. I haven't tried this on linux yet as the problem arose while testing
my app for windows.

--
nosy: +grossetti

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Martina Oefelein]

Changes by Martina Oefelein mart...@oefelein.de:


--
nosy: +oefe

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Benjamin Peterson]

Benjamin Peterson [EMAIL PROTECTED] added the comment:

Any more information on this?

--
nosy: +benjamin.peterson

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Christian Heimes]

Christian Heimes [EMAIL PROTECTED] added the comment:

This popped up on the mailing list today.

Please give this bug a proper review. I *believe* it's fixed but I'm not
sure.

--
nosy: +tiran
priority: normal - critical
type:  - behavior
versions: +Python 2.5, Python 2.6, Python 3.0 -Python 2.4

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Robert Siemer]

Robert Siemer [EMAIL PROTECTED] added the comment:

Update for my comment: python2.5.2 does not show that problem.
python2.4.5 did. I repeat, 2.5.2 does not clean up processes which are
still referenced, but it does clean unreferenced ones.

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Robert Siemer]

Robert Siemer [EMAIL PROTECTED] added the comment:

Bad design stays bad design: the hope that pids don't get reused soon
breaks two assumptions:
1) I don't have to wait() for a child process soon. It's the programs
business.
2) Pids cycle: there are security patches to make pids of future
processes hard to predict by assigning random free pids.

I actually was about to report a bug on subprocess when I bumped into
this one. My problem is pretty thread-less but related:
How do I kill() a process if it's pid got recycled behind my back??

In my opinion the module should work the Python way of doing things,
otherwise programmers will be surprised, as I was:
a) don't do my work (no wait() for things I care of/have reference to)
b) do your work (clean up things I don't care of/have no reference to)

If that's not possible, how does subprocess actually intends to replace
os.spawn*? [Library Reference]

It is still possible to extend the Popen() call to reflect if the caller
wants a P_NOWAITO or P_NOWAIT behavior, but the closer we get to the os,
the less sense does it make to replace some other modules...

--
nosy: +siemer

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Tom Culliton]

Tom Culliton [EMAIL PROTECTED] added the comment:

I'm not sure what the POSIX standards say on this (and MS-Windows may go 
it's own contrary way), but for most real systems the PID is a running 
count (generally 32 bit or larger today) which would have to cycle all 
the way around to repeat.  It's actually done this way on purpose to 
provide the longest possible time between IDs getting reused.  I suspect 
that having it cycle and reuse an ID isn't an issue in practice, and 
keeping a list of results leaves you with the problem of figuring out 
which PID 55367 they're talking about...  Not to mention that if you're 
leaving child process results unchecked for long enough for the PID 
counter to cycle, there are other problems with the application. ;-)

Gregory P. Smith wrote:
 Gregory P. Smith [EMAIL PROTECTED] added the comment:

 Basically it's OK to collect
 all the child exit codes if you record the results and return them when
 requested. This would mean that waitpid and the like would have to check
 a cached list of PIDs and exit statuses before actually waiting.

 note that this would still have problems.  PIDs are recycled by the OS
 so as soon as you've waited on one and the process dies, the OS is free
 to launch a new process using it.  If the new process happens to be
 another one of ours launched by subprocess that would be a problem. 
 Make the cached map of pids - exit codes be a map of pids - [list of
 exit codes] instead?

 _
 Tracker [EMAIL PROTECTED]
 http://bugs.python.org/issue1731717
 _


_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Jean-Paul Calderone]

Jean-Paul Calderone [EMAIL PROTECTED] added the comment:

Which system uses a 32 bit PID?  Not Linux, or FreeBSD, or OS X - they
all use 16 bit PIDs.

--
nosy: +exarkun

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Tom Culliton]

Tom Culliton [EMAIL PROTECTED] added the comment:

AIX, HP-UX, Solaris, 64 bit Linux, ...  Even in the Linux x86 header 
files there's a mix of int and short.  The last time I had to do the 
math on how long it would take the PID to cycle was probably on an AIX 
box and it was a very long time.

Jean-Paul Calderone wrote:
 Jean-Paul Calderone [EMAIL PROTECTED] added the comment:

 Which system uses a 32 bit PID?  Not Linux, or FreeBSD, or OS X - they
 all use 16 bit PIDs.

 --
 nosy: +exarkun

 _
 Tracker [EMAIL PROTECTED]
 http://bugs.python.org/issue1731717
 _


_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Jean-Paul Calderone]

Jean-Paul Calderone [EMAIL PROTECTED] added the comment:

PIDs cycle quite frequently.  You must be thinking of something else. 
Even 64 bit Linux has a maximum PID of 2 ** 15 - 1.

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Gregory P. Smith]

Gregory P. Smith [EMAIL PROTECTED] added the comment:

fwiw, I see pids cycle in a reasonable amount of time on modern
production linux systems today.  its a fact of life, we should deal with
it. :)

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Gregory P. Smith]

Gregory P. Smith [EMAIL PROTECTED] added the comment:

Basically it's OK to collect
all the child exit codes if you record the results and return them when
requested. This would mean that waitpid and the like would have to check
a cached list of PIDs and exit statuses before actually waiting.

note that this would still have problems.  PIDs are recycled by the OS
so as soon as you've waited on one and the process dies, the OS is free
to launch a new process using it.  If the new process happens to be
another one of ours launched by subprocess that would be a problem. 
Make the cached map of pids - exit codes be a map of pids - [list of
exit codes] instead?

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Gregory P. Smith]

Gregory P. Smith [EMAIL PROTECTED] added the comment:

I am unable to reproduce this problem in today's trunk (2.6a) on OSX 10.4.

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Gregory P. Smith]

Changes by Gregory P. Smith:


--
nosy: +gregory.p.smith

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Tom Culliton]

Tom Culliton added the comment:

Looking at the subprocess.py code it occurred to me that it never checks
if the value of self.pid returned by os.fork is -1, this would mean that
later it runs waitpid with -1 as the first argument, putting it into
promiscuous (wait for any process in the group) mode.  This seems like a
bad idea...

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Tom Culliton]

Tom Culliton added the comment:

Good question.  The documentation I was reading was mute on the subject 
so I made a reasonable guess.  Does it throw an exception instead?

Guido van Rossum wrote:
 Guido van Rossum added the comment:

   
 Looking at the subprocess.py code it occurred to me that it never
 checks if the value of self.pid returned by os.fork is -1
 

 What makes you think os.fork(0 can return -1? It's not C you know...

 _
 Tracker [EMAIL PROTECTED]
 http://bugs.python.org/issue1731717
 _


_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Guido van Rossum]

Guido van Rossum added the comment:

 Looking at the subprocess.py code it occurred to me that it never
 checks if the value of self.pid returned by os.fork is -1

What makes you think os.fork(0 can return -1? It's not C you know...

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Guido van Rossum]

Guido van Rossum added the comment:

Yes, like all system calls in the os module.

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Tom Culliton]

Tom Culliton added the comment:

This or some variant also shows up with scons
(http://scons.tigris.org/issues/show_bug.cgi?id=1839) leading to some
nasty intermittent build failures.  Neal may remember how we addressed
this for a Process class in a past life.  Basically it's OK to collect
all the child exit codes if you record the results and return them when
requested. This would mean that waitpid and the like would have to check
a cached list of PIDs and exit statuses before actually waiting.  Don't
know if that's possible/any help in this case or not...

Traceback (most recent call last):
  File /usr/lib/scons-0.97.0d20070918/SCons/Taskmaster.py, line 194,
in execute
self.targets[0].build()
  File /usr/lib/scons-0.97.0d20070918/SCons/Node/__init__.py, line
365, in build
stat = apply(executor, (self,), kw)
  File /usr/lib/scons-0.97.0d20070918/SCons/Executor.py, line 139, in
__call__
return self.do_execute(target, kw)
  File /usr/lib/scons-0.97.0d20070918/SCons/Executor.py, line 129, in
do_execute
status = apply(act, (self.targets, self.sources, env), kw)
  File /usr/lib/scons-0.97.0d20070918/SCons/Action.py, line 332, in
__call__
stat = self.execute(target, source, env)
  File /usr/lib/scons-0.97.0d20070918/SCons/Action.py, line 479, in
execute
result = spawn(shell, escape, cmd_line[0], cmd_line, ENV)
  File /usr/lib/scons-0.97.0d20070918/SCons/Platform/posix.py, line
104, in spawnvpe_spawn
return exec_spawnvpe([sh, '-c', string.join(args)], env)
  File /usr/lib/scons-0.97.0d20070918/SCons/Platform/posix.py, line
68, in exec_spawnvpe
stat = os.spawnvpe(os.P_WAIT, l[0], l, env)
  File /sw/pkgs/python-2.3.5_00/lib/python2.3/os.py, line 553, in spawnvpe
return _spawnvef(mode, file, args, env, execvpe)
  File /sw/pkgs/python-2.3.5_00/lib/python2.3/os.py, line 504, in
_spawnvef
wpid, sts = waitpid(pid, 0)
OSError: [Errno 10] No child processes
scons: building terminated because of errors.

--
nosy: +tom_culliton

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1731717] race condition in subprocess module

[Guido van Rossum]

Guido van Rossum added the comment:

See http://bugs.python.org/issue1236 for a good repeatable testcase.

_
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1731717
_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com