[issue38091] Import deadlock detection causes deadlock

[Victor Zhestkov]

Victor Zhestkov  added the comment:

I ported the fix from 
https://github.com/python/cpython/commit/6daa37fd42c5d5300172728e8b4de74fe0b319fc
 for 3.6 and 3.8 shipped with SLE 15SP2 and openSUSE Tumbleweed, but it seems 
that this fix doesn't help.
I have a deadlocks on running `salt-api` process managing `salt-ssh` systems 
with high workload. The service can get the deadlock in first 5 minutes or 
after 3-60 minutes of running the service with the same workload with almost 
equal chances.

Here is the part of py-bt I see each time:

(gdb) py-bt
Traceback (most recent call first):
  File "", line 107, in acquire
  File "", line 158, in __enter__
  File "", line 595, in _exec
  File "", line 271, in _load_module_shim
  File "", line 852, in load_module
  File "", line 1027, in load_module
  File "", line 1034, in 
_check_name_wrapper
  File "/usr/lib/python3.8/site-packages/salt/loader.py", line 4779, in 
_load_module
  File "/usr/lib/python3.8/site-packages/salt/loader.py", line 1926, in 
_inner_load
if self._load_module(name) and key in self._dict:
  File "/usr/lib/python3.8/site-packages/salt/loader.py", line 2193, in _load
  File "/usr/lib/python3.8/site-packages/salt/utils/lazy.py", line 99, in 
__getitem__
if self._load(key):
  File "/usr/lib/python3.8/site-packages/salt/loader.py", line 1283, in 
__getitem__
func = super().__getitem__(item)
  File "/usr/lib/python3.8/site-packages/salt/loader.py", line 1139, in 
__getitem__
return self._dict[key + self.suffix]
  File "/usr/lib/python3.8/site-packages/salt/template.py", line 495, in 
check_render_pipe_str
  File "/usr/lib/python3.8/site-packages/salt/loader.py", line 1428, in render
f_noext,
  File "/usr/lib/python3.8/site-packages/salt/pillar/__init__.py", line 781, in 
__init__
...

--
nosy: +vzhestkov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38091] Import deadlock detection causes deadlock

[miss-islington]

miss-islington  added the comment:


New changeset 6daa37fd42c5d5300172728e8b4de74fe0b319fc by Armin Rigo in branch 
'master':
bpo-38091: Import deadlock detection causes deadlock (GH-17518)
https://github.com/python/cpython/commit/6daa37fd42c5d5300172728e8b4de74fe0b319fc


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38091] Import deadlock detection causes deadlock

[Armin Rigo]

Change by Armin Rigo :


--
keywords: +patch
pull_requests: +16996
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17518

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38091] Import deadlock detection causes deadlock

[Phil Connell]

Change by Phil Connell :


--
nosy: +pconnell

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38091] Import deadlock detection causes deadlock

[Eric Snow]

Change by Eric Snow :


--
nosy: +brett.cannon, eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38091] Import deadlock detection causes deadlock

[Eric Snow]

Change by Eric Snow :


--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38091] Import deadlock detection causes deadlock

[Ronan Lamy]

New submission from Ronan Lamy :

There seems to be a race condition in importlib._bootstrap._ModuleLock that can 
cause a deadlock. The sequence of operations is as follows:

* Thread 1 calls lock.acquire()
* Thread 1 sets itself as lock.owner and begins importing the module
* Thread 2 calls lock.acquire() and waits for lock.lock
* Thread 2 gets lock.lock
* Thread 1 calls lock.acquire() again, due to a nested import
* Thread 1 sets itself as blocking on lock: _blocking_on[tid1] = lock
* Thread 2 enters lock.has_deadlock()
* Thread 2 busy-waits forever in has_deadlock() because lock.owner == tid1 and 
_blocking_on[tid1] == lock
* Thread 1 waits forever for lock.lock since thread 2 owns it

The issue was found in pypy3 but it also affects all the recent CPython 
versions I tried.
I can reliably reproduce the issue by adding an artificial delay to 
_ModuleLock.has_deadlock(), e.g. with this patch:

diff --git a/Lib/test/test_import/__init__.py b/Lib/test/test_import/__init__.py
index f167c84..7f7188e 100644
--- a/Lib/test/test_import/__init__.py
+++ b/Lib/test/test_import/__init__.py
@@ -435,10 +435,15 @@ class ImportTests(unittest.TestCase):
 os.does_not_exist
 
 def test_concurrency(self):
+def delay_has_deadlock(frame, event, arg):
+if event == 'call' and frame.f_code.co_name == 'has_deadlock':
+time.sleep(0.2)
+
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'data'))
 try:
 exc = None
 def run():
+sys.settrace(delay_has_deadlock)
 event.wait()
 try:
 import package

--
components: Library (Lib)
messages: 351647
nosy: Ronan.Lamy
priority: normal
severity: normal
status: open
title: Import deadlock detection causes deadlock
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com