potiuk opened a new pull request, #48989:
URL: https://github.com/apache/airflow/pull/48989
Wheel package released 8th of April (0.46.1) broke installation of
sdist-only packages (such as pyspark) in our main images. After investigation,
it turned out that this is a combination of things:
* The new wheel package does not inject `bdist_wheel` command to setuptools
any more. This is because setuptools > 70.1 have the bdist_wheel command
vendored in and they do not need wheel at all to build the wheels.
* Python official bookworm-slim images that we use as base images
- includingthe latest versions released today - have old and sometimes
inconsistent version of wheel and setuptools:
Python 3.9: wheel: 0.45.1, setuptools: 58.1.0
Python 3.10: wheel: 0.46.1, setuptools: 65.5.1
Python 3.11: wheel: 0.46.1, setuptools: 65.5.1
Python 3.12: no wheel, no setuptools
This means that there are the following scenarios that trigger the error
that `bdist_wheel` command is missing:
* Python 3.9 -> upgrading wheel (Which we did accidentally by other package)
and not upgrading setuptools -> error
* Python 3.10, 3.11 -> just `pip install pyspark` -> error
* Python 3.12 -> all good, latest setuptools would be installed, no wheel
needed
As a remediation, we are upgrading setuptools to latest released version in
our images. We also fix them - similarly to `pip` and `uv` in order to protect
against any cases where just upgrading to new setuptools might break thigns (As
it happened recently where setuptools upgrade suddenly broke a lot of sdist
packages)
Our automated pre-commits upgrading installers have been modified to also
upgrade setuptools.
See https://github.com/pypa/wheel/issues/660#issuecomment-2788333871 for
details
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
