New submission from Jens Henrik Goebbert <j.goebb...@fz-juelich.de>:
While using [Xpra](https://github.com/Xpra-org/xpra) we came across a bug which might be a Python or a NumPy issue. Perhaps some of you can help us understanding some internals. Calling `import numpy` at the same time in two different threads of the Python program can lead to a race-condition. This happens for example with Xpra when loading the encoder nvjpeg: ``` 2022-03-20 12:54:59,298 cannot load enc_nvjpeg (nvjpeg encoder) Traceback (most recent call last): File "<pythondir>/lib/python3.9/site-packages/xpra/codecs/loader.py", line 52, in codec_import_check ic = __import__(class_module, {}, {}, classnames) File "xpra/codecs/nvjpeg/encoder.pyx", line 8, in init xpra.codecs.nvjpeg.encoder File "<pythondir>/lib/python3.9/site-packages/numpy/__init__.py", line 150, in <module> from . import core File "<pythondir>/lib/python3.9/site-packages/numpy/core/__init__.py", line 51, in <module> del os.environ[envkey] File "<pythondir>/lib/python3.9/os.py", line 695, in __delitem__ raise KeyError(key) from None KeyError: 'OPENBLAS_MAIN_FREE' ``` Here the environment variable OPENBLAS_MAIN_FREE is set in the `numpy` code: https://github.com/numpy/numpy/blob/maintenance/1.21.x/numpy/core/__init__.py#L18 and short after that it is deleted https://github.com/numpy/numpy/blob/maintenance/1.21.x/numpy/core/__init__.py#L51 But this deletion fails ... perhaps because the initialization runs at the same time in two threads :thinking: Shouldn't Python protect us by design? @seberg comments [HERE](https://github.com/numpy/numpy/issues/21223#issuecomment-1074008386): ``` So, my current hypothesis (and I have briefly checked the Python code) is that Python does not do manual locking. But it effectively locks due to this going into C and thus holding the GIL. But somewhere during the import of NumPy, NumPy probably releases the GIL briefly and that could allow the next thread to go into the import machinery. [..] NumPy may be doing some worse than typical stuff here, but right now it seems to me that Python should be protecting us. ``` Can anyone comment on this? ---------- components: Interpreter Core messages: 415687 nosy: jhgoebbert priority: normal severity: normal status: open title: No protection: `import numpy` in two different threads can lead to race-condition type: behavior versions: Python 3.11 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue47082> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
