This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b0e28f8af165 [SPARK-57393][BUILD][FOLLOWUP] Clean out old PySpark
sdists
b0e28f8af165 is described below
commit b0e28f8af16544d8063146c3438a1ab51dc7e304
Author: Nicholas Chammas <[email protected]>
AuthorDate: Wed Jul 22 15:53:35 2026 +0000
[SPARK-57393][BUILD][FOLLOWUP] Clean out old PySpark sdists
### What changes were proposed in this pull request?
Clear stale PySpark sdist tarballs from `python/dist/` before building new
ones in `dev/make-distribution.sh`.
### Why are the changes needed?
#56453 added a post-build validation that checks every
`dist/pyspark*.tar.gz` for top-level LICENSE and NOTICE files. However, if
`python/dist/` already contains tarballs from a previous build, the glob picks
them up and the validation fails with:
```
ERROR: dist/pyspark-4.0.0.dev0.tar.gz is missing LICENSE at the package root
```
The script already cleans `pyspark.egg-info` to avoid stale caches. This
change also cleans out old sdists.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
I reproduced the failure locally with a stale `pyspark-4.0.0.dev0.tar.gz`
in `python/dist/`, applied the fix, and confirmed `make-distribution.sh --pip`
completes successfully.
### Was this patch authored or co-authored using generative AI tooling?
Co-authored with GitHub Copilot.
Closes #56996 from nchammas/SPARK-57393-clean-old-sdists.
Authored-by: Nicholas Chammas <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
---
dev/make-distribution.sh | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index e3bedf15e565..075136238437 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -272,8 +272,10 @@ cp -r "$SPARK_HOME/data" "$DISTDIR"
if [ "$MAKE_PIP" == "true" ]; then
echo "Building python distribution package"
pushd "$SPARK_HOME/python" > /dev/null
- # Delete the egg info file if it exists, this can cache older setup files.
- rm -rf pyspark.egg-info || echo "No existing egg info file, skipping
deletion"
+ # Delete build artifacts that may be left over from earlier builds so that
the build
+ # validation logic below doesn't trip on them.
+ rm -rf pyspark.egg-info
+ rm -f dist/pyspark*.tar.gz
# Ship the Apache LICENSE and NOTICE inside the PySpark source distributions
# (see MANIFEST.in). These are removed again after the sdists are built.
#
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
