This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 992eb15d631a [SPARK-57130][BUILD][FOLLOWUP] make-distribution.sh
copies the generated pyspark.zip into the distribution
992eb15d631a is described below
commit 992eb15d631a7e2cea53dce3b3dfe441963e67d3
Author: Cheng Pan <[email protected]>
AuthorDate: Mon Aug 31 23:10:00 2026 +0800
[SPARK-57130][BUILD][FOLLOWUP] make-distribution.sh copies the generated
pyspark.zip into the distribution
### What changes were proposed in this pull request?
`make-distribution.sh` copies `python/lib/pyspark.zip` into the
distribution, alongside the git-tracked python files.
### Why are the changes needed?
SPARK-57130 (#56186) made `make-distribution.sh` copy only git-tracked
files for the `python` folder. `python/lib/pyspark.zip` is generated by the
build (assembly module) and gitignored, so it no longer reaches the
distribution.
Without it:
- `PythonUtils.sparkPythonPath` adds `$SPARK_HOME/python/lib/pyspark.zip`
to `PYTHONPATH`, so `bin/pyspark` and Python applications cannot import pyspark
from the tarball unless it is pip-installed.
- PySpark applications in YARN mode fail at `Client.findPySparkArchives()`,
which requires the file to exist.
### Does this PR introduce _any_ user-facing change?
No. SPARK-57130 broke some PySpark use cases of the binary distribution,
but it is unreleased, so no users are affected.
### How was this patch tested?
Ran `dev/make-distribution.sh` on a fresh worktree before and after
SPARK-57130, and compared the resulting `dist/python` trees file by file: they
are identical except for `python/lib/pyspark.zip`, which is missing after
SPARK-57130. Exercised the patched copy step against the built tree, and
`dist/python/lib/pyspark.zip` is restored:
```
$ ls dist/python/lib
PY4J_LICENSE.txt py4j-0.10.9.9-src.zip pyspark.zip
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex CLI
Closes #58428 from pan3793/SPARK-57130-followup.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 6b28d92c1ead40fb912ab3a9496a933e6c51842a)
Signed-off-by: Cheng Pan <[email protected]>
---
dev/make-distribution.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 075136238437..4a2bd0991012 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -367,6 +367,7 @@ cp "$SPARK_HOME/README.md" "$DISTDIR"
cp -r "$SPARK_HOME/bin" "$DISTDIR"
if command -v git && command -v cpio && git rev-parse --git-dir 2>/dev/null;
then
git ls-files -z "$SPARK_HOME/python" | cpio -0pdm "$DISTDIR"
+ cp "$SPARK_HOME/python/lib/pyspark.zip" "$DISTDIR/python/lib"
else
cp -r "$SPARK_HOME/python" "$DISTDIR"
fi
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
