Taragolis opened a new issue, #38988: URL: https://github.com/apache/airflow/issues/38988
### Body Our test with build example docker image (https://airflow.apache.org/docs/docker-stack/build.html#example-when-you-add-packages-requiring-compilation) started failed recently with build error ```console #6 12.80 Building wheels for collected packages: mpi4py #6 12.80 Building wheel for mpi4py (pyproject.toml): started #6 13.71 Building wheel for mpi4py (pyproject.toml): finished with status 'error' #6 13.74 error: subprocess-exited-with-error #6 13.74 #6 13.74 × Building wheel for mpi4py (pyproject.toml) did not run successfully. #6 13.74 │ exit code: 1 #6 13.74 ╰─> [212 lines of output] #6 13.74 running bdist_wheel #6 13.74 running build #6 13.74 running build_src #6 13.74 running build_py #6 13.74 creating build #6 13.74 creating build/lib.linux-x86_64-cpython-38 #6 13.74 creating build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/run.py -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/__init__.py -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/bench.py -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/__main__.py -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 creating build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/_base.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/pool.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/_core.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/server.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/__init__.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/aplus.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/__main__.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/_lib.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 creating build/lib.linux-x86_64-cpython-38/mpi4py/util #6 13.74 copying src/mpi4py/util/dtlib.py -> build/lib.linux-x86_64-cpython-38/mpi4py/util #6 13.74 copying src/mpi4py/util/__init__.py -> build/lib.linux-x86_64-cpython-38/mpi4py/util #6 13.74 copying src/mpi4py/util/pkl5.py -> build/lib.linux-x86_64-cpython-38/mpi4py/util #6 13.74 copying src/mpi4py/MPI.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/run.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/__main__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/bench.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/__init__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/dl.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/py.typed -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/__init__.pxd -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/MPI.pxd -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 copying src/mpi4py/libmpi.pxd -> build/lib.linux-x86_64-cpython-38/mpi4py #6 13.74 creating build/lib.linux-x86_64-cpython-38/mpi4py/include #6 13.74 creating build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py #6 13.74 copying src/mpi4py/include/mpi4py/mpi4py.MPI.h -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py #6 13.74 copying src/mpi4py/include/mpi4py/mpi4py.MPI_api.h -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py #6 13.74 copying src/mpi4py/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py #6 13.74 copying src/mpi4py/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py #6 13.74 copying src/mpi4py/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py #6 13.74 copying src/mpi4py/futures/__main__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/_core.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/server.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/__init__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/pool.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/_lib.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/futures/aplus.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures #6 13.74 copying src/mpi4py/util/pkl5.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/util #6 13.74 copying src/mpi4py/util/dtlib.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/util #6 13.74 copying src/mpi4py/util/__init__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/util #6 13.74 running build_clib #6 13.74 MPI configuration: [mpi] from 'mpi.cfg' #6 13.74 MPI C compiler: /usr/bin/mpicc #6 13.74 MPI C++ compiler: /usr/bin/mpicxx #6 13.74 MPI F compiler: /usr/bin/mpifort #6 13.74 MPI F90 compiler: /usr/bin/mpif90 #6 13.74 MPI F77 compiler: /usr/bin/mpif77 #6 13.74 checking for library 'lmpe' ... #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o #6 13.74 /usr/bin/mpicc _configtest.o -llmpe -o _configtest #6 13.74 /usr/bin/ld: cannot find -llmpe: No such file or directory #6 13.74 collect2: error: ld returned 1 exit status #6 13.74 failure. #6 13.74 removing: _configtest.c _configtest.o #6 13.74 building 'mpe' dylib library #6 13.74 creating build/temp.linux-x86_64-cpython-38 #6 13.74 creating build/temp.linux-x86_64-cpython-38/src #6 13.74 creating build/temp.linux-x86_64-cpython-38/src/lib-pmpi #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-cpython-38/src/lib-pmpi/mpe.o #6 13.74 creating build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi #6 13.74 /usr/bin/mpicc -shared -Wl,--no-as-needed build/temp.linux-x86_64-cpython-38/src/lib-pmpi/mpe.o -o build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi/libmpe.so #6 13.74 checking for library 'vt-mpi' ... #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o #6 13.74 /usr/bin/mpicc _configtest.o -lvt-mpi -o _configtest #6 13.74 /usr/bin/ld: cannot find -lvt-mpi: No such file or directory #6 13.74 collect2: error: ld returned 1 exit status #6 13.74 failure. #6 13.74 removing: _configtest.c _configtest.o #6 13.74 checking for library 'vt.mpi' ... #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o #6 13.74 /usr/bin/mpicc _configtest.o -lvt.mpi -o _configtest #6 13.74 /usr/bin/ld: cannot find -lvt.mpi: No such file or directory #6 13.74 collect2: error: ld returned 1 exit status #6 13.74 failure. #6 13.74 removing: _configtest.c _configtest.o #6 13.74 building 'vt' dylib library #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c src/lib-pmpi/vt.c -o build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt.o #6 13.74 /usr/bin/mpicc -shared -Wl,--no-as-needed build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt.o -o build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi/libvt.so #6 13.74 checking for library 'vt-mpi' ... #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o #6 13.74 /usr/bin/mpicc _configtest.o -lvt-mpi -o _configtest #6 13.74 /usr/bin/ld: cannot find -lvt-mpi: No such file or directory #6 13.74 collect2: error: ld returned 1 exit status #6 13.74 failure. #6 13.74 removing: _configtest.c _configtest.o #6 13.74 checking for library 'vt.mpi' ... #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o #6 13.74 /usr/bin/mpicc _configtest.o -lvt.mpi -o _configtest #6 13.74 /usr/bin/ld: cannot find -lvt.mpi: No such file or directory #6 13.74 collect2: error: ld returned 1 exit status #6 13.74 failure. #6 13.74 removing: _configtest.c _configtest.o #6 13.74 building 'vt-mpi' dylib library #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c src/lib-pmpi/vt-mpi.c -o build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt-mpi.o #6 13.74 /usr/bin/mpicc -shared -Wl,--no-as-needed build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt-mpi.o -o build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi/libvt-mpi.so #6 13.74 checking for library 'vt-hyb' ... #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o #6 13.74 /usr/bin/mpicc _configtest.o -lvt-hyb -o _configtest #6 13.74 /usr/bin/ld: cannot find -lvt-hyb: No such file or directory #6 13.74 collect2: error: ld returned 1 exit status #6 13.74 failure. #6 13.74 removing: _configtest.c _configtest.o #6 13.74 checking for library 'vt.ompi' ... #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o #6 13.74 /usr/bin/mpicc _configtest.o -lvt.ompi -o _configtest #6 13.74 /usr/bin/ld: cannot find -lvt.ompi: No such file or directory #6 13.74 collect2: error: ld returned 1 exit status #6 13.74 failure. #6 13.74 removing: _configtest.c _configtest.o #6 13.74 building 'vt-hyb' dylib library #6 13.74 /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c src/lib-pmpi/vt-hyb.c -o build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt-hyb.o #6 13.74 /usr/bin/mpicc -shared -Wl,--no-as-needed build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt-hyb.o -o build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi/libvt-hyb.so #6 13.74 running build_ext #6 13.74 MPI configuration: [mpi] from 'mpi.cfg' #6 13.74 MPI C compiler: /usr/bin/mpicc #6 13.74 MPI C++ compiler: /usr/bin/mpicxx #6 13.74 MPI F compiler: /usr/bin/mpifort #6 13.74 MPI F90 compiler: /usr/bin/mpif90 #6 13.74 MPI F77 compiler: /usr/bin/mpif77 #6 13.74 checking for dlopen() availability ... #6 13.74 checking for header 'dlfcn.h' ... #6 13.74 gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/airflow/.local/include -I/usr/local/include/python3.8 -c _configtest.c -o _configtest.o #6 13.74 success! #6 13.74 removing: _configtest.c _configtest.o #6 13.74 success! #6 13.74 checking for library 'dl' ... #6 13.74 gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/airflow/.local/include -I/usr/local/include/python3.8 -c _configtest.c -o _configtest.o #6 13.74 Traceback (most recent call last): #6 13.74 File "/home/airflow/.local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module> #6 13.74 main() #6 13.74 File "/home/airflow/.local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main #6 13.74 json_out['return_val'] = hook(**hook_input['kwargs']) #6 13.74 File "/home/airflow/.local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel #6 13.74 return _build_backend().build_wheel(wheel_directory, config_settings, #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 410, in build_wheel #6 13.74 return self._build_with_temp_dir( #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir #6 13.74 self.run_setup() #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup #6 13.74 exec(code, locals()) #6 13.74 File "<string>", line 644, in <module> #6 13.74 File "<string>", line 641, in main #6 13.74 File "<string>", line 492, in run_setup #6 13.74 File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 541, in setup #6 13.74 return fcn_setup(**attrs) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 104, in setup #6 13.74 return distutils.core.setup(**attrs) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 184, in setup #6 13.74 return run_commands(dist) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 200, in run_commands #6 13.74 dist.run_commands() #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands #6 13.74 self.run_command(cmd) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command #6 13.74 super().run_command(command) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command #6 13.74 cmd_obj.run() #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 368, in run #6 13.74 self.run_command("build") #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command #6 13.74 self.distribution.run_command(command) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command #6 13.74 super().run_command(command) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command #6 13.74 cmd_obj.run() #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run #6 13.74 self.run_command(cmd_name) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command #6 13.74 self.distribution.run_command(command) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command #6 13.74 super().run_command(command) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command #6 13.74 cmd_obj.run() #6 13.74 File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line [1213](https://github.com/apache/airflow/actions/runs/8674524298/job/23787271307#step:9:1214), in run #6 13.74 cmd_build_ext.build_ext.run(self) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 91, in run #6 13.74 _build_ext.run(self) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run #6 13.74 self.build_extensions() #6 13.74 File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 1241, in build_extensions #6 13.74 self.build_extension(ext) #6 13.74 File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 1272, in build_extension #6 13.74 self.config_extension(ext) #6 13.74 File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 1255, in config_extension #6 13.74 configure(ext, config_cmd) #6 13.74 File "<string>", line 188, in configure_dl #6 13.74 File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 607, in check_library #6 13.74 ok = self.try_link(body, headers, include_dirs, #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/config.py", line 246, in try_link #6 13.74 self._link(body, headers, include_dirs, libraries, library_dirs, lang) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/config.py", line 138, in _link #6 13.74 self.compiler.link_executable( #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/ccompiler.py", line 781, in link_executable #6 13.74 self.link( #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/unixccompiler.py", line 233, in link #6 13.74 lib_opts = gen_lib_options(self, library_dirs, runtime_library_dirs, libraries) #6 13.74 File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/ccompiler.py", line [1232](https://github.com/apache/airflow/actions/runs/8674524298/job/23787271307#step:9:1233), in gen_lib_options #6 13.74 lib_opts.extend(always_iterable(compiler.runtime_library_dir_option(dir))) #6 13.74 File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 68, in rpath_option #6 13.74 if option.startswith('-R'): #6 13.74 AttributeError: 'list' object has no attribute 'startswith' #6 13.74 [end of output] #6 13.74 ``` We should resolve this issue, potentially the solution might be appear into the upstream issue task: - https://github.com/mpi4py/mpi4py/issues/484 ### Committer - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
