Francesco Macagno created SPARK-51595:
-----------------------------------------
Summary: pyspark pip install fails with latest setuptools (78.x)
Key: SPARK-51595
URL: https://issues.apache.org/jira/browse/SPARK-51595
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 3.5.5
Environment: pyenv virtualenv and docker-based python both ran into
same error.
Reporter: Francesco Macagno
setuptools 78.0.1 broke installing pyspark by no longer allowing
`description-file` in setup.cfg. Thankfully, they have since reverted the
change, but it seems odd that pyspark still has that issue given it seemed to
have been fixed a while ago.
```
Collecting pyspark==3.5.5 (from -r requirements.txt (line 18))
Using cached pyspark-3.5.5.tar.gz (317.2 MB)
Running command pip subprocess to install build dependencies
Using pip 25.0.1 from
/Users/fmacagno/.pyenv/versions/3.9.21/envs/airflow/lib/python3.9/site-packages/pip
(python 3.9)
Collecting setuptools>=40.8.0
Obtaining dependency information for setuptools>=40.8.0 from
https://files.pythonhosted.org/packages/42/c8/3faed884acdb2c1f2eb353cbacdd1ee4943de89a199d1f622ebefb6170e5/setuptools-78.0.1-py3-none-any.whl.metadata
Using cached setuptools-78.0.1-py3-none-any.whl.metadata (6.6 kB)
Using cached setuptools-78.0.1-py3-none-any.whl (1.3 MB)
Installing collected packages: setuptools
Successfully installed setuptools-78.0.1
Installing build dependencies ... done
Running command Getting requirements to build wheel
Traceback (most recent call last):
File
"/Users/fmacagno/.pyenv/versions/3.9.21/envs/airflow/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py",
line 389, in <module>
main()
File
"/Users/fmacagno/.pyenv/versions/3.9.21/envs/airflow/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py",
line 373, in main
json_out["return_val"] = hook(**hook_input["kwargs"])
File
"/Users/fmacagno/.pyenv/versions/3.9.21/envs/airflow/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py",
line 143, in get_requires_for_build_wheel
return hook(config_settings)
File
"/private/var/folders/3x/02mhw__978919nbqh8z01r400000gn/T/pip-build-env-dxr75ldx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py",
line 334, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
File
"/private/var/folders/3x/02mhw__978919nbqh8z01r400000gn/T/pip-build-env-dxr75ldx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py",
line 304, in _get_build_requires
self.run_setup()
File
"/private/var/folders/3x/02mhw__978919nbqh8z01r400000gn/T/pip-build-env-dxr75ldx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py",
line 522, in run_setup
super().run_setup(setup_script=setup_script)
File
"/private/var/folders/3x/02mhw__978919nbqh8z01r400000gn/T/pip-build-env-dxr75ldx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py",
line 320, in run_setup
exec(code, locals())
File "<string>", line 225, in <module>
File
"/private/var/folders/3x/02mhw__978919nbqh8z01r400000gn/T/pip-build-env-dxr75ldx/overlay/lib/python3.9/site-packages/setuptools/__init__.py",
line 116, in setup
_install_setup_requires(attrs)
File
"/private/var/folders/3x/02mhw__978919nbqh8z01r400000gn/T/pip-build-env-dxr75ldx/overlay/lib/python3.9/site-packages/setuptools/__init__.py",
line 87, in _install_setup_requires
dist.parse_config_files(ignore_option_errors=True)
File
"/private/var/folders/3x/02mhw__978919nbqh8z01r400000gn/T/pip-build-env-dxr75ldx/overlay/lib/python3.9/site-packages/setuptools/dist.py",
line 730, in parse_config_files
self._parse_config_files(filenames=inifiles)
File
"/private/var/folders/3x/02mhw__978919nbqh8z01r400000gn/T/pip-build-env-dxr75ldx/overlay/lib/python3.9/site-packages/setuptools/dist.py",
line 599, in _parse_config_files
opt = self._enforce_underscore(opt, section)
File
"/private/var/folders/3x/02mhw__978919nbqh8z01r400000gn/T/pip-build-env-dxr75ldx/overlay/lib/python3.9/site-packages/setuptools/dist.py",
line 629, in _enforce_underscore
raise InvalidConfigError(
setuptools.errors.InvalidConfigError: Invalid dash-separated key
'description-file' in 'metadata' (setup.cfg), please use the underscore name
'description_file' instead.
error: subprocess-exited-with-error
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
