[issue31070] test_threaded_import: KeyError ignored in _get_module_lock..cb on AMD64 Windows8.1 Refleaks 3.x

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I think the solution can be simpler. PR 3033 uses the global import lock for 
guarding modification of the _module_locks dictionary.

--
components: +Interpreter Core -Tests
stage:  -> patch review
type:  -> behavior
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31070] test_threaded_import: KeyError ignored in _get_module_lock..cb on AMD64 Windows8.1 Refleaks 3.x

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +3068

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31070] test_threaded_import: KeyError ignored in _get_module_lock..cb on AMD64 Windows8.1 Refleaks 3.x

[STINNER Victor]

STINNER Victor added the comment:

I wrote https://github.com/python/cpython/pull/3029 which seems to fix the 
issue.

I don't know importlib well enough to understand why we need a weak reference 
to a lock.

My PR adds a second lock per module lock (!) to be able to wait until 
_module_locks[name] is deleted when a thread detects that _module_locks[name] 
exists and the associated lock was destroyed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31070] test_threaded_import: KeyError ignored in _get_module_lock..cb on AMD64 Windows8.1 Refleaks 3.x

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +3064

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31070] test_threaded_import: KeyError ignored in _get_module_lock..cb on AMD64 Windows8.1 Refleaks 3.x

[STINNER Victor]

STINNER Victor added the comment:

The problem is the weakref callback of _module_locks. It seems like the 
callback is called after a second thread replaced _module_locks[name]

* Thread 1: _get_module_lock('random') sets _module_locks['random'] and 
registers a weakref callback on the newly created lock

* Thread 1: the lock is destroyed, the weakref callback will be called as soon 
as possible

* Thread 2: _get_module_lock('random') sees that the weakref is dead and so 
replaces _module_locks['random'] with a new lock, and registers a new weakref 
callback on the second lock

* Thread 1: Call the weakref callback removing _module_locks['random'] -- BUG: 
the callback is called too late, after Thread 2 already sets a new lock

* Thread 2: The lock is destroyed, the second weakref callback will be called 
as soon as possible

* Thread 2: The second callback is called and fails with KeyError since 
_module_locks['random'] was already called in the Thread 1 in the meanwhile

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31070] test_threaded_import: KeyError ignored in _get_module_lock..cb on AMD64 Windows8.1 Refleaks 3.x

[STINNER Victor]

STINNER Victor added the comment:

Oh, it's easy to reproduce the bug on the master branch. Example on Linux:

haypo@selma$ ./python -m test test_threaded_import
Run tests sequentially
0:00:00 load avg: 0.16 [1/1] test_threaded_import
Exception ignored in: .cb at 0x7f53201e2f70>
Traceback (most recent call last):
  File "", line 176, in cb
KeyError: ('random',)
1 test OK.

Total duration: 2 sec
Tests result: SUCCESS

--
nosy: +brett.cannon, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31070] test_threaded_import: KeyError ignored in _get_module_lock..cb on AMD64 Windows8.1 Refleaks 3.x

[STINNER Victor]

New submission from STINNER Victor:

See bpo-bpo-30891 and bpo-30876 for recent changes in importlib.

http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.x/builds/57/steps/test/logs/stdio

1:33:30 [312/406/2] test_threaded_import passed (30 sec) -- running: 
test_decimal (826 sec), test_set (84 sec), test_mmap (1151 sec)
beginning 6 repetitions
123456
...Exception ignored in: .cb at 
0x00813CA18EB8>
Traceback (most recent call last):
  File "", line 176, in cb
KeyError: ('random',)
...

--
components: Tests
messages: 299393
nosy: haypo, serhiy.storchaka
priority: normal
severity: normal
status: open
title: test_threaded_import: KeyError ignored in _get_module_lock..cb 
on AMD64 Windows8.1 Refleaks 3.x
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com