[issue1731717] race condition in subprocess module

STINNER Victor added the comment: "Hi, Could anyone help here to identify in which Python release the bug is fixed. I am unable to deduce from the bug tracker interface in which release this issue is fixed." Oh, this is an old issue :-) msg123946: "r87233 fixes the OSError escaping from wait(

[issue1731717] race condition in subprocess module

Krishna Oza added the comment: Hi, Could anyone help here to identify in which Python release the bug is fixed. I am unable to deduce from the bug tracker interface in which release this issue is fixed. Regards. -- nosy: +Krishna Oza ___ Python tra

[issue1731717] race condition in subprocess module

Vitaly added the comment: Is this issue fixed in python 2.7.10? I am experiencing strange race conditions in code that combines threads with multiprocessing pool, whereby a thread is spawned by each pool worker. Running on latest Mac OS X. -- nosy: +vitaly

[issue1731717] race condition in subprocess module

jan matejek added the comment: please check my logic here, but the patched code seems to throw away perfectly valid return codes: in wait(), self._handle_exitstatus(sts) gets called unconditionally, and it resets self.returncode also unconditionally. now, if a _cleanup() already did _internal_

[issue1731717] race condition in subprocess module

Antoine Pitrou added the comment: It seems the canonical spelling is SIGCHLD. SIGCLD doesn't exist everywhere and it produces test failures under OS X: http://www.python.org/dev/buildbot/all/builders/AMD64%20Leopard%203.x http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%2

[issue1731717] race condition in subprocess module

Gregory P. Smith added the comment: sorry, i meant 3.2beta2 above. release27-maint: r87234 targeting 2.7.2 release31-maint: r87235 targeting 3.1.4 -- resolution: -> fixed status: open -> closed ___ Python tracker

[issue1731717] race condition in subprocess module

Gregory P. Smith added the comment: r87233 fixes the OSError escaping from wait() issue when SIGCLD is set to be ignored. (to appear in 3.2beta1; it is a candidate for backporting to 3.1 and 2.7) -- ___ Python tracker

[issue1731717] race condition in subprocess module

James Lamanna added the comment: stubbing out subprocess._cleanup does not work around the problem from this example on 2.6.5: import subprocess, signal subprocess._cleanup = lambda: None signal.signal(signal.SIGCLD, signal.SIG_IGN) subprocess.Popen(['echo','foo']).wait() ja...@hyla:~$ pytho

[issue1731717] race condition in subprocess module

Gregory P. Smith added the comment: A workaround for those still having problems with this: stub out subprocess._cleanup with a no-op method. It it only useful if your app is ever using subprocess and forgetting to call wait() on Popen objects before they are deleted. If you are, you can ke

[issue1731717] race condition in subprocess module

Changes by Guido van Rossum : -- nosy: -gvanrossum ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue1731717] race condition in subprocess module

Changes by Jonas H. : -- nosy: +jonash ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org

[issue1731717] race condition in subprocess module

red added the comment: I'm using an old Plone/Zope Product, PHParser, that uses the popen2 call ... same problem for me. Is there a thread-safe alternative to execute subprocesses in threads? I need a patch!!! thanks in advance!!! -- nosy: +shaphiro __

[issue1731717] race condition in subprocess module

Changes by Collin Winter : -- nosy: +collinwinter, jyasskin ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: h

[issue1731717] race condition in subprocess module

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: htt

[issue1731717] race condition in subprocess module

Stefan added the comment: I have exactly the same problem. Is there a thread-safe alternative to execute subprocesses in threads? -- nosy: +santoni ___ Python tracker ___ ___

[issue1731717] race condition in subprocess module

Yonas added the comment: Gregory, Awesome! Approx. how long until we hear back from you in this report? -- ___ Python tracker ___

[issue1731717] race condition in subprocess module

Gregory P. Smith added the comment: I really don't care about the age of a bug. This bug is young. I've fixed many bugs over twice its age in the past. Regardless, I've got some serious subprocess internal refactoring changes coming in the very near future to explicitly deal with thread saf

[issue1731717] race condition in subprocess module

Yonas added the comment: By the way, in three months from today, this bug will be 3 years old. -- ___ Python tracker ___ ___ Py

[issue1731717] race condition in subprocess module

Yonas added the comment: Ben Finney's comment suggests to me that this bug is being ignored. Am I wrong? "with extra notes in the documentation that anyone using child processes needs to be wary of signal handling." Why should they be wary? We should just fix this bug. -- __

[issue1731717] race condition in subprocess module

Yonas added the comment: http://groups.google.com/group/comp.lang.python/browse_thread/thread/9a853d0308c8e55a "I'm also glad to see a test case that causes exactly the same error with or without the presence of a ‘daemon.DaemonContext’. Further research shows that handling of ‘SIGCLD’ (or ‘

[issue1731717] race condition in subprocess module

Changes by Florent Xicluna : -- keywords: -buildbot ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue1731717] race condition in subprocess module

Florent Xicluna added the comment: >>> import subprocess, signal >>> signal.signal(signal.SIGCLD, signal.SIG_IGN) 0 >>> subprocess.Popen(['echo','foo']).wait() foo Traceback (most recent call last): File "", line 1, in File "./Lib/subprocess.py", line 1229, in wait pid, sts = _eintr_ret

[issue1731717] race condition in subprocess module

Yonas added the comment: Florent, Have you tested any of the sample test programs mentioned in this bug report? For example, the one by Joel Martin (kanaka). I'd suggest to test those first before marking this issue as fixed. - Yonas -- status: pending -> open _

[issue1731717] race condition in subprocess module

Florent Xicluna added the comment: Previous buildbot failures were in test_multiprocessing: http://bugs.python.org/issue1731717#msg100430 Now it should be fixed: - r78777, r78787, r78790 on 2.x - r78798 on 3.x -- resolution: -> fixed stage: test needed -> committed/rejected status:

[issue1731717] race condition in subprocess module

Florent Xicluna added the comment: Sometimes IA64 Ubuntu bot fails on this one. http://www.python.org/dev/buildbot/all/builders/ia64%20Ubuntu%20trunk/builds/571 -- keywords: +buildbot ___ Python tracker

[issue1731717] race condition in subprocess module

Joel Martin added the comment: I can reproduce the problem (or at least get the same symptom) by doing this (in 2.4.6, 2.5.4 and 2.6.2): import subprocess, signal signal.signal(signal.SIGCLD, signal.SIG_IGN) subprocess.Popen(['echo','foo']).wait() The echo command completes, but the subprocess

[issue1731717] race condition in subprocess module

R. David Murray added the comment: I'm afraid I'm not going to be installing exim in order to test this. Perhaps someone else will be interested enough to do so, perhaps not :) Copying files into /usr/lib/pythonx.x is a very odd thing to do, by the way (though it should have no impact on the b

[issue1731717] race condition in subprocess module

Yonas added the comment: I'm assuming that exim4 is reading config files from /etc/exim4/conf.d/*. To make sure, you can enforce "split file mode" by running `sudo dpkg-reconfigure exim4-config`. It should be one of the last questions present to you. -- ___

[issue1731717] race condition in subprocess module

Yonas added the comment: Also, copy exim_local_scan2.py to /usr/lib/python2.6/ -- Added file: http://bugs.python.org/file14134/exim_local_scan2.py ___ Python tracker ___ _

[issue1731717] race condition in subprocess module

Yonas added the comment: To test with exim4 is easy. To reproduce on Ubuntu Jaunty: 1. apt-get install exim4-daemon-heavy 2. echo "local_scan = /usr/lib/exim4/local_scan/libpyexim.so" > /etc/exim4/conf.d/main/15_py-exim_plugin_path 3. cd /usr/lib/exim4/local_scan 4. Compile mylib, output wil

[issue1731717] race condition in subprocess module

R. David Murray added the comment: I have confirmed that none of the original test cases referenced here or in the referenced issues fails on python 2.5.4, 2.6.2, trunk, or py3k on linux. The test case attached to this ticket I cannot test as I don't understand how one would run it (and exim ap

[issue1731717] race condition in subprocess module

Changes by Yonas : Added file: http://bugs.python.org/file14087/main.c ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue1731717] race condition in subprocess module

Yonas added the comment: I always get a subprocess error when using embedded python 2.6.2: File "/usr/lib/python2.6/subprocess.py", line 1123, in wait: pid, sts = os.waitpid(self.pid, 0): OSError: [Errno 10] No child processes Example library and main program are attached. -- nosy

[issue1731717] race condition in subprocess module

Changes by djc : -- nosy: +djc ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman

[issue1731717] race condition in subprocess module

Gabriel added the comment: I had this happen to me today, I used Popen and os.waitpid(p.pid, 0) and I got an "OSError: [Errno 10] No child processes". I haven't tried adding a sleep, since it's unreliable. I'm using python 2.5.2, Windows XP. I haven't tried this on linux yet as the problem arose

[issue1731717] race condition in subprocess module

Changes by Martina Oefelein : -- nosy: +oefe ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue1731717] race condition in subprocess module

Benjamin Peterson <[EMAIL PROTECTED]> added the comment: Any more information on this? -- nosy: +benjamin.peterson ___ Python tracker <[EMAIL PROTECTED]> ___ ___

[issue1731717] race condition in subprocess module

Christian Heimes <[EMAIL PROTECTED]> added the comment: This popped up on the mailing list today. Please give this bug a proper review. I *believe* it's fixed but I'm not sure. -- nosy: +tiran priority: normal -> critical type: -> behavior versions: +Python 2.5, Python 2.6, Python 3.0

[issue1731717] race condition in subprocess module

Robert Siemer <[EMAIL PROTECTED]> added the comment: Update for my comment: python2.5.2 does not show that problem. python2.4.5 did. I repeat, 2.5.2 does not clean up processes which are still referenced, but it does clean unreferenced ones. _ Tracker <[EMAIL

[issue1731717] race condition in subprocess module

Robert Siemer <[EMAIL PROTECTED]> added the comment: Bad design stays bad design: the hope that pids don't get reused soon breaks two assumptions: 1) I don't have to wait() for a child process soon. It's the programs business. 2) Pids cycle: there are security patches to make pids of future proce

[issue1731717] race condition in subprocess module

Gregory P. Smith <[EMAIL PROTECTED]> added the comment: fwiw, I see pids cycle in a reasonable amount of time on modern production linux systems today. its a fact of life, we should deal with it. :) _ Tracker <[EMAIL PROTECTED]>

[issue1731717] race condition in subprocess module

Jean-Paul Calderone <[EMAIL PROTECTED]> added the comment: PIDs cycle quite frequently. You must be thinking of something else. Even 64 bit Linux has a maximum PID of 2 ** 15 - 1. _ Tracker <[EMAIL PROTECTED]> __

[issue1731717] race condition in subprocess module

Tom Culliton <[EMAIL PROTECTED]> added the comment: AIX, HP-UX, Solaris, 64 bit Linux, ... Even in the Linux x86 header files there's a mix of int and short. The last time I had to do the math on how long it would take the PID to cycle was probably on an AIX box and it was a very long time.

[issue1731717] race condition in subprocess module

Jean-Paul Calderone <[EMAIL PROTECTED]> added the comment: Which system uses a 32 bit PID? Not Linux, or FreeBSD, or OS X - they all use 16 bit PIDs. -- nosy: +exarkun _ Tracker <[EMAIL PROTECTED]> __

[issue1731717] race condition in subprocess module

Tom Culliton <[EMAIL PROTECTED]> added the comment: I'm not sure what the POSIX standards say on this (and MS-Windows may go it's own contrary way), but for most real systems the PID is a running count (generally 32 bit or larger today) which would have to cycle all the way around to repeat.

[issue1731717] race condition in subprocess module

Gregory P. Smith <[EMAIL PROTECTED]> added the comment: I am unable to reproduce this problem in today's trunk (2.6a) on OSX 10.4. _ Tracker <[EMAIL PROTECTED]> _ ___

[issue1731717] race condition in subprocess module

Gregory P. Smith <[EMAIL PROTECTED]> added the comment: """Basically it's OK to collect all the child exit codes if you record the results and return them when requested. This would mean that waitpid and the like would have to check a cached list of PIDs and exit statuses before actually waiting.

[issue1731717] race condition in subprocess module

Changes by Gregory P. Smith: -- nosy: +gregory.p.smith _ Tracker <[EMAIL PROTECTED]> _ ___ Python-bugs-list mailing list Unsubscribe

[issue1731717] race condition in subprocess module

Guido van Rossum added the comment: Yes, like all system calls in the os module. _ Tracker <[EMAIL PROTECTED]> _ ___ Python-bugs-list mailing

[issue1731717] race condition in subprocess module

Tom Culliton added the comment: Good question. The documentation I was reading was mute on the subject so I made a "reasonable" guess. Does it throw an exception instead? Guido van Rossum wrote: > Guido van Rossum added the comment: > > >> Looking at the subprocess.py code it occurred to m

[issue1731717] race condition in subprocess module

Guido van Rossum added the comment: > Looking at the subprocess.py code it occurred to me that it never > checks if the value of self.pid returned by os.fork is -1 What makes you think os.fork(0 can return -1? It's not C you know... _ Tracker <[EMAIL PROTECTE

[issue1731717] race condition in subprocess module

Tom Culliton added the comment: Looking at the subprocess.py code it occurred to me that it never checks if the value of self.pid returned by os.fork is -1, this would mean that later it runs waitpid with -1 as the first argument, putting it into promiscuous (wait for any process in the group) mo

[issue1731717] race condition in subprocess module

Tom Culliton added the comment: This or some variant also shows up with scons (http://scons.tigris.org/issues/show_bug.cgi?id=1839) leading to some nasty intermittent build failures. Neal may remember how we addressed this for a Process class in a past life. Basically it's OK to collect all the

[issue1731717] race condition in subprocess module

Guido van Rossum added the comment: See http://bugs.python.org/issue1236 for a good repeatable testcase. _ Tracker <[EMAIL PROTECTED]> _ ___ P