This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 56840d30ca31 [SPARK-57393] Build: PySpark and SparkR source
distributions are missing LICENSE and NOTICE files
56840d30ca31 is described below
commit 56840d30ca3101474c2cc1eafe8642cc57b2f209
Author: Huaxin Gao <[email protected]>
AuthorDate: Thu Jun 11 15:59:10 2026 -0700
[SPARK-57393] Build: PySpark and SparkR source distributions are missing
LICENSE and NOTICE files
### What changes were proposed in this pull request?
Make the PySpark and SparkR source distributions include top-level LICENSE
and NOTICE files:
- Add include LICENSE / include NOTICE to python/MANIFEST.in.
- In dev/make-distribution.sh, copy LICENSE/NOTICE into python/ and R/pkg/
before building the sdists (with an EXIT trap to clean them up).
- The classic pyspark sdist bundles the assembly jars, so it ships the
binary license variants (LICENSE-binary/NOTICE-binary plus the full
licenses-binary set), mirroring the binary distribution. The pyspark_connect,
pyspark_client, and SparkR artifacts bundle no jars, so they ship the plain
source LICENSE/NOTICE. packaging/classic/setup.py falls back to licenses/ when
licenses-binary/ is absent (RELEASE-mode builds).
- During the SparkR build, + file LICENSE is added to DESCRIPTION
temporarily (restored after the build) so R CMD check --as-cran does not warn
that the bundled LICENSE is not mentioned. The committed DESCRIPTION is
unchanged, so SparkR CI is unaffected. The "Non-standard file/directory found
at top level: 'NOTICE'" NOTE cannot be silenced this way and is expected in
release-build logs.
- Add tar-listing regression guards after the sdist and SparkR builds that
fail the release build if LICENSE or NOTICE ever goes missing again, instead of
relying on an RC vote to catch it.
### Why are the changes needed?
ASF release policy requires every distributed artifact to include LICENSE
and NOTICE. The pyspark, pyspark_connect, pyspark_client, and SparkR source
tarballs currently don't, which was raised as a -1 during the Spark 4.2.0 RC1
vote.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Built the source distributions locally and confirmed each contains LICENSE
and NOTICE at the package root
### Was this patch authored or co-authored using generative AI tooling?
Yes, drafted with assistance from Cursor.
Closes #56453 from huaxingao/fix_license.
Authored-by: Huaxin Gao <[email protected]>
Signed-off-by: huaxin-gao_snow <[email protected]>
(cherry picked from commit b88952cabfff54973957cdc21d3a548acf951ac1)
Signed-off-by: huaxin-gao_snow <[email protected]>
---
R/.gitignore | 4 +++
dev/make-distribution.sh | 69 +++++++++++++++++++++++++++++++++++++++
python/.gitignore | 3 ++
python/MANIFEST.in | 4 ++-
python/packaging/classic/setup.py | 11 +++++--
5 files changed, 88 insertions(+), 3 deletions(-)
diff --git a/R/.gitignore b/R/.gitignore
index c98504ab0778..693b2b7c6917 100644
--- a/R/.gitignore
+++ b/R/.gitignore
@@ -6,3 +6,7 @@ pkg/man
pkg/html
SparkR.Rcheck/
SparkR_*.tar.gz
+# Transient copies made by dev/make-distribution.sh while building the package.
+pkg/LICENSE
+pkg/NOTICE
+DESCRIPTION.orig
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 16607e45ae65..242b43b1419a 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -32,6 +32,21 @@ set -x
SPARK_HOME="$(cd "`dirname "$0"`/.."; pwd)"
DISTDIR="$SPARK_HOME/dist"
+# The Apache LICENSE and NOTICE are copied into the Python and R package
+# directories below so they are bundled into the source distributions. Remove
+# them on exit so a failed build does not leave stray files behind.
+function cleanup_dist_license_files {
+ rm -f "$SPARK_HOME/python/LICENSE" "$SPARK_HOME/python/NOTICE" \
+ "$SPARK_HOME/R/pkg/LICENSE" "$SPARK_HOME/R/pkg/NOTICE"
+ # Restore the SparkR DESCRIPTION if a release build patched it in place (see
+ # the R packaging section). Guards against an interrupted build leaving the
+ # tracked DESCRIPTION modified.
+ if [ -f "$SPARK_HOME/R/DESCRIPTION.orig" ]; then
+ mv -f "$SPARK_HOME/R/DESCRIPTION.orig" "$SPARK_HOME/R/pkg/DESCRIPTION"
+ fi
+}
+trap cleanup_dist_license_files EXIT
+
MAKE_TGZ=false
MAKE_PIP=false
MAKE_R=false
@@ -253,9 +268,39 @@ if [ "$MAKE_PIP" == "true" ]; then
pushd "$SPARK_HOME/python" > /dev/null
# Delete the egg info file if it exists, this can cache older setup files.
rm -rf pyspark.egg-info || echo "No existing egg info file, skipping
deletion"
+ # Ship the Apache LICENSE and NOTICE inside the PySpark source distributions
+ # (see MANIFEST.in). These are removed again after the sdists are built.
+ #
+ # The classic pyspark sdist bundles the assembly jars
(packaging/classic/setup.py
+ # builds a deps/jars symlink farm), so it ships the binary LICENSE/NOTICE
that
+ # enumerate the bundled third-party jars' licenses, mirroring the binary
+ # distribution above. The connect and client sdists bundle no jars and ship
the
+ # plain source LICENSE/NOTICE.
+ if [ -e "$SPARK_HOME/LICENSE-binary" ]; then
+ cp "$SPARK_HOME/LICENSE-binary" LICENSE
+ cp "$SPARK_HOME/NOTICE-binary" NOTICE
+ else
+ cp "$SPARK_HOME/LICENSE" LICENSE
+ cp "$SPARK_HOME/NOTICE" NOTICE
+ fi
python3 packaging/classic/setup.py sdist
+
+ cp "$SPARK_HOME/LICENSE" LICENSE
+ cp "$SPARK_HOME/NOTICE" NOTICE
python3 packaging/connect/setup.py sdist
python3 packaging/client/setup.py sdist
+ rm -f LICENSE NOTICE
+
+ # Guard against regressions: every PySpark sdist must contain LICENSE and
NOTICE
+ # at the package root. The missing files were only caught by a Spark 4.2.0
RC1
+ # vote -1 (SPARK-57393); fail the release build here instead of at vote time.
+ for f in dist/pyspark*.tar.gz; do
+ listing=$(tar tzf "$f")
+ for required in LICENSE NOTICE; do
+ grep -qE "^[^/]+/$required\$" <<< "$listing" || \
+ { echo "ERROR: $f is missing $required at the package root"; exit 1; }
+ done
+ done
popd > /dev/null
else
echo "Skipping building python distribution package"
@@ -266,9 +311,33 @@ if [ "$MAKE_R" == "true" ]; then
echo "Building R source package"
R_PACKAGE_VERSION=`grep Version "$SPARK_HOME/R/pkg/DESCRIPTION" | awk
'{print $NF}'`
pushd "$SPARK_HOME/R" > /dev/null
+ # Ship the Apache LICENSE and NOTICE inside the SparkR source package. These
+ # are removed again after the package is built.
+ cp "$SPARK_HOME/LICENSE" pkg/LICENSE
+ cp "$SPARK_HOME/NOTICE" pkg/NOTICE
+ # Reference the bundled LICENSE from DESCRIPTION so `R CMD check --as-cran`
does
+ # not emit "File LICENSE is not mentioned in the DESCRIPTION file". The
committed
+ # DESCRIPTION is left untouched because SparkR CI runs check-cran.sh without
the
+ # LICENSE file present; this edit is transient and restored after the build
(and
+ # by the EXIT trap on failure). The backup lives outside pkg/ so R CMD check
does
+ # not flag it as a non-standard file. NOTE: the "Non-standard file 'NOTICE'"
note
+ # cannot be silenced this way and is expected.
+ cp pkg/DESCRIPTION "$SPARK_HOME/R/DESCRIPTION.orig"
+ sed 's/^License: Apache License (== 2.0)$/License: Apache License (== 2.0) +
file LICENSE/' \
+ "$SPARK_HOME/R/DESCRIPTION.orig" > pkg/DESCRIPTION
# Build source package and run full checks
# Do not source the check-cran.sh - it should be run from where it is for it
to set SPARK_HOME
NO_TESTS=1 "$SPARK_HOME/R/check-cran.sh"
+ mv -f "$SPARK_HOME/R/DESCRIPTION.orig" pkg/DESCRIPTION
+ rm -f pkg/LICENSE pkg/NOTICE
+
+ # Guard against regressions: the SparkR source package must contain LICENSE
and
+ # NOTICE at the package root (SPARK-57393).
+ listing=$(tar tzf "SparkR_$R_PACKAGE_VERSION.tar.gz")
+ for required in LICENSE NOTICE; do
+ grep -qE "^[^/]+/$required\$" <<< "$listing" || \
+ { echo "ERROR: SparkR source package is missing $required"; exit 1; }
+ done
# Move R source package to match the Spark release version if the versions
are not the same.
# NOTE(shivaram): `mv` throws an error on Linux if source and destination
are same file
diff --git a/python/.gitignore b/python/.gitignore
index 6d95b545fd56..b865a73038ce 100644
--- a/python/.gitignore
+++ b/python/.gitignore
@@ -7,3 +7,6 @@ build/
dist/
./setup.py
./setup.cfg
+# Transient copies made by dev/make-distribution.sh while building the sdists.
+/LICENSE
+/NOTICE
diff --git a/python/MANIFEST.in b/python/MANIFEST.in
index 45c9dca8b474..82979a0344c3 100644
--- a/python/MANIFEST.in
+++ b/python/MANIFEST.in
@@ -21,10 +21,12 @@ recursive-include deps/jars *.jar
graft deps/bin
recursive-include deps/sbin spark-config.sh spark-daemon.sh
start-history-server.sh stop-history-server.sh
recursive-include deps/data *.data *.txt
-recursive-include deps/licenses *.txt
+graft deps/licenses
recursive-include deps/examples *.py
recursive-include lib *.zip
include README.md
+include LICENSE
+include NOTICE
# Note that these commands are processed in the order they appear, so keep
# this exclude at the end.
diff --git a/python/packaging/classic/setup.py
b/python/packaging/classic/setup.py
index 661ba5a8a7ed..1b13dba32c9e 100755
--- a/python/packaging/classic/setup.py
+++ b/python/packaging/classic/setup.py
@@ -104,7 +104,14 @@ EXAMPLES_PATH = os.path.join(SPARK_HOME,
"examples/src/main/python")
SCRIPTS_PATH = os.path.join(SPARK_HOME, "bin")
USER_SCRIPTS_PATH = os.path.join(SPARK_HOME, "sbin")
DATA_PATH = os.path.join(SPARK_HOME, "data")
-LICENSES_PATH = os.path.join(SPARK_HOME, "licenses")
+# The classic PySpark package bundles the assembly jars, so it ships the binary
+# license texts (licenses-binary), which enumerate those jars' licenses,
mirroring
+# the binary distribution. The connect/client packages bundle no jars.
+LICENSES_PATH = os.path.join(SPARK_HOME, "licenses-binary")
+if not os.path.isdir(LICENSES_PATH):
+ # In a binary release dist (the RELEASE mode below), the binary license
texts
+ # were already copied to licenses/ (see dev/make-distribution.sh).
+ LICENSES_PATH = os.path.join(SPARK_HOME, "licenses")
SCRIPTS_TARGET = os.path.join(TEMP_PATH, "bin")
USER_SCRIPTS_TARGET = os.path.join(TEMP_PATH, "sbin")
@@ -337,7 +344,7 @@ try:
],
"pyspark.python.lib": ["*.zip"],
"pyspark.data": ["*.txt", "*.data"],
- "pyspark.licenses": ["*.txt"],
+ "pyspark.licenses": ["*"],
"pyspark.examples.src.main.python": ["*.py", "*/*.py"],
},
scripts=scripts,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
