New submission from Michał Górny <mgo...@gentoo.org>: There is a race condition in distutils' build_ext implementation. When the same source file is used to build multiple extensions, distutils attempts to build it multiple times using the same output file, in parallel. This means that the link editor can grab the file while another compiler instance is overwriting it. The results vary from compile errors to cryptic dyld failures when attempting to load the module.
I've created a trivial reproducer that I've attached in a patch form. For convenience, it's also available on my GitHub: https://github.com/mgorny/distutils-build_ext-race The reproducer consists of two extension modules sharing the same file. The race.sh script attempts to build the extension and then import it. The process is repeated until something fails, e.g.: + python3.10 setup.py build_ext -i -j4 running build_ext building 'bar' extension creating build building 'foo' extension creating build/temp.linux-x86_64-3.10 creating build/temp.linux-x86_64-3.10 x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c bar.c -o build/temp.linux-x86_64-3.10/bar.o x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c foo.c -o build/temp.linux-x86_64-3.10/foo.o x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/foo.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/bar.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/bar.cpython-310-x86_64-linux-gnu.so + python3.10 -c 'import foo; import bar' Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so: undefined symbol: call_shared + echo 'Reproduced at iteration 256' Reproduced at iteration 256 + break ---------- components: Distutils files: 0001-A-reproducer-for-distutils-build_ext-race-condition.patch keywords: patch messages: 388410 nosy: dstufft, eric.araujo, mgorny priority: normal severity: normal status: open title: Race conditions when the same source file used to build mutliple extensions type: compile error versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9 Added file: https://bugs.python.org/file49863/0001-A-reproducer-for-distutils-build_ext-race-condition.patch _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue43459> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
