This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new a187555a8b9e [SPARK-58167][SPARK-57393][3.5][BUILD] Build: PySpark and
SparkR source distributions are missing LICENSE and NOTICE files
a187555a8b9e is described below
commit a187555a8b9eb53538cdf99c7eddf18b6398e309
Author: Holden Karau <[email protected]>
AuthorDate: Wed Aug 12 11:47:51 2026 -0700
[SPARK-58167][SPARK-57393][3.5][BUILD] Build: PySpark and SparkR source
distributions are missing LICENSE and NOTICE files
Changes proposed:
Backport of
[[SPARK-57393](https://issues.apache.org/jira/browse/SPARK-57393)] Build:
PySpark and SparkR source distributions are missing LICENSE and NOTICE files
Make the PySpark and SparkR source distributions include top-level LICENSE
and NOTICE files:
- Add include LICENSE / include NOTICE to python/MANIFEST.in.
- In dev/make-distribution.sh, copy LICENSE/NOTICE into python/ and R/pkg/
before building the sdists (with an EXIT trap to clean them up).
- The classic pyspark sdist bundles the assembly jars, so it ships the
binary license variants (LICENSE-binary/NOTICE-binary plus the full
licenses-binary set), mirroring the binary distribution. The pyspark_connect,
pyspark_client, and SparkR artifacts bundle no jars, so they ship the plain
source LICENSE/NOTICE. packaging/classic/setup.py falls back to licenses/ when
licenses-binary/ is absent (RELEASE-mode builds).
- During the SparkR build, + file LICENSE is added to DESCRIPTION
temporarily (restored after the build) so R CMD check --as-cran does not warn
that the bundled LICENSE is not mentioned. The committed DESCRIPTION is
unchanged, so SparkR CI is unaffected. The "Non-standard file/directory found
at top level: 'NOTICE'" NOTE cannot be silenced this way and is expected in
release-build logs.
- Add tar-listing regression guards after the sdist and SparkR builds that
fail the release build if LICENSE or NOTICE ever goes missing again, instead of
relying on an RC vote to catch it.
Why are they needed?
ASF release policy requires every distributed artifact to include LICENSE
and NOTICE. The pyspark, pyspark_connect, pyspark_client, and SparkR source
tarballs currently don't, which was raised as a -1 during the Spark 3.5.9 RC1
No
Built the source distributions locally and confirmed each contains LICENSE
and NOTICE at the package root
Closes #57427 from holdenk/SPARK-58167-add-license-files-r2.
Lead-authored-by: Holden Karau <[email protected]>
Co-authored-by: Claude <Holden's Claude Robot [email protected]>
Co-authored-by: Holden Karau <[email protected]>
Co-authored-by: Huaxin Gao <[email protected]>
Signed-off-by: Holden Karau <[email protected]>
---
R/.gitignore | 4 +++
dev/make-distribution.sh | 63 ++++++++++++++++++++++++++++++++++++++++++++++++
python/.gitignore | 3 +++
python/MANIFEST.in | 8 +++++-
python/setup.py | 11 +++++++--
5 files changed, 86 insertions(+), 3 deletions(-)
diff --git a/R/.gitignore b/R/.gitignore
index c98504ab0778..693b2b7c6917 100644
--- a/R/.gitignore
+++ b/R/.gitignore
@@ -6,3 +6,7 @@ pkg/man
pkg/html
SparkR.Rcheck/
SparkR_*.tar.gz
+# Transient copies made by dev/make-distribution.sh while building the package.
+pkg/LICENSE
+pkg/NOTICE
+DESCRIPTION.orig
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index ef7c010e930a..1b6210a6570c 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -32,6 +32,21 @@ set -x
SPARK_HOME="$(cd "`dirname "$0"`/.."; pwd)"
DISTDIR="$SPARK_HOME/dist"
+# The Apache LICENSE and NOTICE are copied into the Python and R package
+# directories below so they are bundled into the source distributions. Remove
+# them on exit so a failed build does not leave stray files behind.
+function cleanup_dist_license_files {
+ rm -f "$SPARK_HOME/python/LICENSE" "$SPARK_HOME/python/NOTICE" \
+ "$SPARK_HOME/R/pkg/LICENSE" "$SPARK_HOME/R/pkg/NOTICE"
+ # Restore the SparkR DESCRIPTION if a release build patched it in place (see
+ # the R packaging section). Guards against an interrupted build leaving the
+ # tracked DESCRIPTION modified.
+ if [ -f "$SPARK_HOME/R/DESCRIPTION.orig" ]; then
+ mv -f "$SPARK_HOME/R/DESCRIPTION.orig" "$SPARK_HOME/R/pkg/DESCRIPTION"
+ fi
+}
+trap cleanup_dist_license_files EXIT
+
MAKE_TGZ=false
MAKE_PIP=false
MAKE_R=false
@@ -240,7 +255,31 @@ if [ "$MAKE_PIP" == "true" ]; then
pushd "$SPARK_HOME/python" > /dev/null
# Delete the egg info file if it exists, this can cache older setup files.
rm -rf pyspark.egg-info || echo "No existing egg info file, skipping
deletion"
+ # Ship the Apache LICENSE and NOTICE inside the PySpark source distribution
+ # (see MANIFEST.in). These are removed again after the sdist is built.
+ #
+ # The pyspark sdist bundles the assembly jars, so it ships the binary
+ # LICENSE/NOTICE that enumerate the bundled third-party jars' licenses,
+ # mirroring the binary distribution above.
+ if [ -e "$SPARK_HOME/LICENSE-binary" ]; then
+ cp "$SPARK_HOME/LICENSE-binary" LICENSE
+ cp "$SPARK_HOME/NOTICE-binary" NOTICE
+ else
+ cp "$SPARK_HOME/LICENSE" LICENSE
+ cp "$SPARK_HOME/NOTICE" NOTICE
+ fi
python3 setup.py sdist
+ # Guard against regressions: every PySpark sdist must contain LICENSE and
NOTICE
+ # at the package root. The missing files were only caught by a Spark 4.2.0
RC1
+ # vote -1 (SPARK-57393); fail the release build here instead of at vote time.
+ for f in dist/pyspark*.tar.gz; do
+ listing=$(tar tzf "$f")
+ for required in LICENSE NOTICE; do
+ grep -qE "^[^/]+/$required\$" <<< "$listing" || \
+ { echo "ERROR: $f is missing $required at the package root"; exit 1; }
+ done
+ done
+ rm -f LICENSE NOTICE
popd > /dev/null
else
echo "Skipping building python distribution package"
@@ -251,9 +290,33 @@ if [ "$MAKE_R" == "true" ]; then
echo "Building R source package"
R_PACKAGE_VERSION=`grep Version "$SPARK_HOME/R/pkg/DESCRIPTION" | awk
'{print $NF}'`
pushd "$SPARK_HOME/R" > /dev/null
+ # Ship the Apache LICENSE and NOTICE inside the SparkR source package. These
+ # are removed again after the package is built.
+ cp "$SPARK_HOME/LICENSE" pkg/LICENSE
+ cp "$SPARK_HOME/NOTICE" pkg/NOTICE
+ # Reference the bundled LICENSE from DESCRIPTION so `R CMD check --as-cran`
does
+ # not emit "File LICENSE is not mentioned in the DESCRIPTION file". The
committed
+ # DESCRIPTION is left untouched because SparkR CI runs check-cran.sh without
the
+ # LICENSE file present; this edit is transient and restored after the build
(and
+ # by the EXIT trap on failure). The backup lives outside pkg/ so R CMD check
does
+ # not flag it as a non-standard file. NOTE: the "Non-standard file 'NOTICE'"
note
+ # cannot be silenced this way and is expected.
+ cp pkg/DESCRIPTION "$SPARK_HOME/R/DESCRIPTION.orig"
+ sed 's/^License: Apache License (== 2.0)$/License: Apache License (== 2.0) +
file LICENSE/' \
+ "$SPARK_HOME/R/DESCRIPTION.orig" > pkg/DESCRIPTION
# Build source package and run full checks
# Do not source the check-cran.sh - it should be run from where it is for it
to set SPARK_HOME
NO_TESTS=1 "$SPARK_HOME/R/check-cran.sh"
+ mv -f "$SPARK_HOME/R/DESCRIPTION.orig" pkg/DESCRIPTION
+ rm -f pkg/LICENSE pkg/NOTICE
+
+ # Guard against regressions: the SparkR source package must contain LICENSE
and
+ # NOTICE at the package root (SPARK-57393).
+ listing=$(tar tzf "SparkR_$R_PACKAGE_VERSION.tar.gz")
+ for required in LICENSE NOTICE; do
+ grep -qE "^[^/]+/$required\$" <<< "$listing" || \
+ { echo "ERROR: SparkR source package is missing $required"; exit 1; }
+ done
# Move R source package to match the Spark release version if the versions
are not the same.
# NOTE(shivaram): `mv` throws an error on Linux if source and destination
are same file
diff --git a/python/.gitignore b/python/.gitignore
index 52128cf844a7..8ebf8df2b14d 100644
--- a/python/.gitignore
+++ b/python/.gitignore
@@ -3,3 +3,6 @@ docs/_build/
pyspark.egg-info
build/
dist/
+# Transient copies made by dev/make-distribution.sh while building the sdists.
+/LICENSE
+/NOTICE
diff --git a/python/MANIFEST.in b/python/MANIFEST.in
index 862d62b1d3b2..5faccc246cf0 100644
--- a/python/MANIFEST.in
+++ b/python/MANIFEST.in
@@ -19,8 +19,14 @@ recursive-include deps/jars *.jar
graft deps/bin
recursive-include deps/sbin spark-config.sh spark-daemon.sh
start-history-server.sh stop-history-server.sh
recursive-include deps/data *.data *.txt
-recursive-include deps/licenses *.txt
+graft deps/licenses
recursive-include deps/examples *.py
recursive-include lib *.zip
recursive-include pyspark *.pyi py.typed
include README.md
+include LICENSE
+include NOTICE
+
+# Note that these commands are processed in the order they appear, so keep
+# this exclude at the end.
+global-exclude *.py[cod] __pycache__ .DS_Store
diff --git a/python/setup.py b/python/setup.py
index 16ae358caec6..4b7023b855db 100755
--- a/python/setup.py
+++ b/python/setup.py
@@ -85,7 +85,14 @@ EXAMPLES_PATH = os.path.join(SPARK_HOME,
"examples/src/main/python")
SCRIPTS_PATH = os.path.join(SPARK_HOME, "bin")
USER_SCRIPTS_PATH = os.path.join(SPARK_HOME, "sbin")
DATA_PATH = os.path.join(SPARK_HOME, "data")
-LICENSES_PATH = os.path.join(SPARK_HOME, "licenses")
+# The classic PySpark package bundles the assembly jars, so it ships the binary
+# license texts (licenses-binary), which enumerate those jars' licenses,
mirroring
+# the binary distribution. The connect/client packages bundle no jars.
+LICENSES_PATH = os.path.join(SPARK_HOME, "licenses-binary")
+if not os.path.isdir(LICENSES_PATH):
+ # In a binary release dist (the RELEASE mode below), the binary license
texts
+ # were already copied to licenses/ (see dev/make-distribution.sh).
+ LICENSES_PATH = os.path.join(SPARK_HOME, "licenses")
SCRIPTS_TARGET = os.path.join(TEMP_PATH, "bin")
USER_SCRIPTS_TARGET = os.path.join(TEMP_PATH, "sbin")
@@ -298,7 +305,7 @@ try:
],
"pyspark.python.lib": ["*.zip"],
"pyspark.data": ["*.txt", "*.data"],
- "pyspark.licenses": ["*.txt"],
+ "pyspark.licenses": ["*"],
"pyspark.examples.src.main.python": ["*.py", "*/*.py"],
},
scripts=scripts,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
