cagney added the comment:
So:
#1 we've a bug: the single-threaded ProcessPoolExecutor test program should
work 100% reliably - it does not
#2 we've a cause: ProcessPoolExecutor is implemented internally using an
unfortunate combination of fork and threads, this is causing the deadlock
#3
cagney added the comment:
@gregory.p.smith, I'm puzzled by your references to POSIX and/or os.fork().
The code in question looks like:
import concurrent.futures
import sys
def f():
import ctypes
while True:
with concurrent.futures.ProcessPoolExecutor() as executor:
ftr
cagney added the comment:
We're discussing vanilla Python, for instance v3.7.0 is:
git clone .../cpython
cd cpython
git checkout v3.7.0
./configure --prefix=/home/python/v3.7.0
make -j && make -j install
(my 3.6.x wasn't vanilla, but I clearly stated that)
Like I also m
cagney added the comment:
script to capture stack backtrace at time of fork, last backtrace printed will
be for hang
--
Added file: https://bugs.python.org/file48272/gdb.sh
___
Python tracker
<https://bugs.python.org/issue35
cagney added the comment:
run ProcessPoolExecutor with one fixed child (over ride default of #cores)
--
Added file: https://bugs.python.org/file48271/cf-deadlock-1.py
___
Python tracker
<https://bugs.python.org/issue35
cagney added the comment:
Here's a possible stack taken during the fork():
Thread 1 "python3" hit Breakpoint 1, 0x77124734 in fork () from
/lib64/libc.so.6
Thread 1814 (Thread 0x7fffe69d5700 (LWP 23574)):
#0 0x77bc24e5 in __pthread_mutex_unlock_usercnt () fr
cagney added the comment:
(disclaimer: I'm mashing my high level backtraces in with @jwiki's low level
backtraces)
The Python backtrace shows the deadlocked process called 'f' which then
'called':
import ctypes
which, in turn 'called':
from _ctypes import Union, Structure, Array
cagney added the comment:
@hroncok see comment msg339370
Vanilla 3.7.0 (re-confirmed) didn't contain the change, nor did 3.6.8 (ok, that
isn't vanilla) but both can hang using the test. It can take a while and,
subjectively, it seems to depend on machine load. I've even struggled to get
New submission from cagney :
How can I detect that os.posix_spawn(setsid=True) is available at runtime?
I'd like to use os.posix_spawn(setsid=True) when it is available, and
(assuming I'm getting this right) os.posix_spawn(setpgroup=0) as a poor
fallback.
--
components: IO
messages
New submission from cagney :
When O_CLOEXEC is defined the file is opened with that flag (YA! - this means
that the operation is atomic and, by default, the FD will be closed across
os.posix_spawn()).
However the code then goes on an executes:
#ifndef MS_WINDOWS
if (_Py_set_inheritable
New submission from cagney :
pty.openpty(), on systems with a working os.openpty() / openpty(3) executes:
if (openpty(_fd, _fd, NULL, NULL, NULL) != 0)
goto posix_error;
if (_Py_set_inheritable(master_fd, 0, NULL) < 0)
goto error;
if (_Py_set_inheritable(slave_fd
New submission from cagney :
Buried in issue36533; it should probably be turned into a test case.
Exception ignored in:
Traceback (most recent call last):
File "/home/python/v3.7.3/lib/python3.7/logging/__init__.py", line
269, in _after_at_fork_weak_calls
_at_fork_weak_call
cagney added the comment:
BTW, non-visible change might be to use a global readers-writer lock where
fork() is the writer.
https://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock
--
___
Python tracker
<https://bugs.python.org/issue36
cagney added the comment:
> nope. the contrived emit() pseudocode from msg339650 never defined b in the
> first place. that code, if massaged into python syntax would raise an
> UnboundLocalError no matter who executed it. a fork() from another thread
> woul
cagney added the comment:
Another example is PTY:
Python 2.7.15 (default, Oct 15 2018, 15:24:06)
[GCC 8.1.1 20180712 (Red Hat 8.1.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>
cagney added the comment:
I pointed out two issues with breaking the locks.
Your response addressed the less important issue:
- the guarantee that log records don't interleave is lost
Since the documentation seems to be silent the guarantee (or expectation) is
implied - logging systems
cagney added the comment:
I think the only pragmatic solution here is to add an optional parameter to
logging.basicConfig() that specifies that the logger should use a single global
lock; and then start documenting that thread locks and fork() don't work well
together.
And note
cagney added the comment:
On Fri, 5 Apr 2019 at 05:02, Gregory P. Smith wrote:
>
>
> Gregory P. Smith added the comment:
>
> A stdlib alternative to this whole mess would be to avoid acquiring the
> logging locks before fork() as we currently do and just blindly re-initial
cagney added the comment:
On Fri, 5 Apr 2019 at 04:15, Gregory P. Smith wrote:
>
>
> New submission from Gregory P. Smith :
>
> I'm spawning a dicussion buried in the way too long thread of
> https://bugs.python.org/issue6721 over here into its own specific issue to
> t
cagney added the comment:
Here's the children; yes there are somehow 4 children sitting around.
Hopefully this is enough to figure out where things deadlock.
29970 8752 8752 29970 pts/6 8752 Sl+ 1000 1:00 | | \_
./v3.7.3/bin/python3 cf-deadlock.py
8752 8975 8752 29970 pts
cagney added the comment:
Below is a backtrace from the deadlock.
It happens because the logging code is trying to acquire two per-logger locks;
and in an order different to the random order used by the fork() handler.
The code in question has a custom class DebugHandler(logging.Handler
cagney added the comment:
> acquiring locks before fork in the thread doing the forking and releasing
> them afterwards is always the safe thing to do.
It's also an easy way to cause a deadlock:
- register_at_fork() et.al. will cause per-logger locks to be acquired before
the globa
cagney added the comment:
More info from adding a faulthandler ...
15 def f():
16 import ctypes
17
18 for i in range(0,50):
19 sys.stdout.write("\r%d" % i)
20 sys.stdout.flush()
21 signal.alarm(60*2)
22 for j in ra
cagney added the comment:
> Does your code use any C code that forks on its own without properly calling
> the C Python PyOS_BeforeFork(), PyOS_AfterFork_Parent(), and
> PyOS_AfterFork_Child() APIs?
No.
Is there a web page explaining how to pull a python backtrace from all the
cagney added the comment:
I'm seeing cf-deadlock-alarm.py hangs on vanilla python 3.7.[0123] with:
Linux 5.0.5-100.fc28.x86_64 #1 SMP Wed Mar 27 22:16:29 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
glibc-2.27-37.fc28.x86_64
can anyone reproduce this?
I also wonder if this is connected to bpo
cagney added the comment:
I suspect 3b699932e5ac3e7 is causing a hang in libreswan's kvmrunner.py on
Fedora.
Looking at the Fedora RPMs:
python3-3.7.0-9.fc29.x86_64 didn't contain the fix and worked
python3-3.7.1-4.fc29.x86_64 reverted the fix (for anaconda) and worked
python3-3.7.2-4.fc29
cagney added the comment:
I've attached a variation on cf-deadlock.py that, should nothing happen for 2
minutes, will kill itself. Useful with git bisect.
--
Added file: https://bugs.python.org/file48242/cf-deadlock-alarm.py
___
Python tracker
Change by cagney :
--
nosy: +cagney
___
Python tracker
<https://bugs.python.org/issue35809>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.pyth
Change by cagney :
--
nosy: +cagney
___
Python tracker
<https://bugs.python.org/issue35866>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.pyth
29 matches
Mail list logo
