This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new f2cedafc5b62 [SPARK-58747][PYTHON] Move include commands from
MANIFEST.in into setup.py files
f2cedafc5b62 is described below
commit f2cedafc5b62c33333b90a1dce6457b4dfda4207
Author: Nicholas Chammas <[email protected]>
AuthorDate: Fri Aug 14 19:01:22 2026 -0700
[SPARK-58747][PYTHON] Move include commands from MANIFEST.in into setup.py
files
### What changes were proposed in this pull request?
Move various include commands from MANIFEST.in into the various setup.py
files. MANIFEST.in `recursive-include ... *` is recursive, whereas `*` in
setuptools isn't. The equivalent in setuptools is to use `**`.
Eliminate some of the setup.py hacks that were implemented to work around
MANIFEST.in over-packaging files.
### Why are the changes needed?
Having duplicate (and unnecessary!) packaging specs across setup.py and
MANIFEST.in is [confusing]. MANIFEST.in was also triggering more files to be
included than desired, which required some hacks to work around. With this
cleanup, some of those hacks are no longer necessary.
The new addition of an explicit `license_files` argument is not necessary
-- setuptools packages these files by default -- but nice to have since it's
explicit.
[confusing]:
https://github.com/apache/spark/pull/57763#issuecomment-5262613111
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
I built the three sdists off of master and then again from this branch. I
diffed their entire contents, both files and file contents, and found no
differences except in the MANIFEST.in and setup.py that we changed here, which
is expected.
I also repeated this test but by calling `./dev/make-distribution.sh --pip`
instead of `python ... sdist`, since `make-distribution.sh` stages license and
JAR files before building the sdist. Again, no differences except what we
expect in MANIFEST.in and setup.py.
Here are the test scripts I used:
-
[sdist-diff.sh](https://github.com/user-attachments/files/31071343/sdist-diff.sh)
-
[sdist-diff-make-dist.sh](https://github.com/user-attachments/files/31071338/sdist-diff-make-dist.sh)
Our CI packaging tests are a bit slim. I've separately proposed #57645 to
make them more comprehensive.
### Was this patch authored or co-authored using generative AI tooling?
I wrote the test scripts with assistance from GitHub Copilot.
Closes #57973 from nchammas/pyspark-manifest-cleanup.
Authored-by: Nicholas Chammas <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit dc15294b6550c3016d51e40636ebaf75985a5bc4)
Signed-off-by: Tian Gao <[email protected]>
---
python/MANIFEST.in | 27 ++++++++++++---------------
python/packaging/classic/setup.py | 16 +++++++++-------
python/packaging/client/setup.py | 18 +++++++++---------
python/packaging/connect/setup.py | 19 +++++++------------
4 files changed, 37 insertions(+), 43 deletions(-)
diff --git a/python/MANIFEST.in b/python/MANIFEST.in
index 82979a0344c3..e57d60b8db98 100644
--- a/python/MANIFEST.in
+++ b/python/MANIFEST.in
@@ -14,20 +14,17 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# Reference: https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html
-
-recursive-include pyspark *.pyi py.typed *.json
-recursive-include deps/jars *.jar
-graft deps/bin
-recursive-include deps/sbin spark-config.sh spark-daemon.sh
start-history-server.sh stop-history-server.sh
-recursive-include deps/data *.data *.txt
-graft deps/licenses
-recursive-include deps/examples *.py
-recursive-include lib *.zip
-include README.md
-include LICENSE
-include NOTICE
+# NOTE
+#
+# This file exists only for the exclude below, which applies across every
+# package in every setup.py. (setuptools's `exclude_package_data` is scoped
+# to the individual package and setup.py it's declared in). To include non-
+# `.py` files, use `package_data` in setup.py instead of adding them here.
+#
+# See:
+# - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
+# - https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html
-# Note that these commands are processed in the order they appear, so keep
-# this exclude at the end.
+# Keep this exclude at the end. MANIFEST.in commands are processed in the order
+# they appear.
global-exclude *.py[cod] __pycache__ .DS_Store
diff --git a/python/packaging/classic/setup.py
b/python/packaging/classic/setup.py
index 5ec33f405d49..37f71d1bddcb 100755
--- a/python/packaging/classic/setup.py
+++ b/python/packaging/classic/setup.py
@@ -214,7 +214,7 @@ try:
copyfile("pyspark/shell.py", "pyspark/python/pyspark/shell.py")
if in_spark:
- # !!HACK ALTERT!!
+ # !!HACK ALERT!!
# `setup.py` has to be located with the same directory with the
package.
# Therefore, we copy the current file, and place it at `spark/python`
directory.
# After that, we remove it in the end.
@@ -340,8 +340,9 @@ try:
"pyspark.examples.src.main.python": "deps/examples",
},
package_data={
- "pyspark.jars": ["*.jar"],
- "pyspark.bin": ["*"],
+ "pyspark": ["**/*.pyi", "**/py.typed", "**/*.json"],
+ "pyspark.jars": ["**/*.jar"],
+ "pyspark.bin": ["**/*"],
"pyspark.sbin": [
"spark-config.sh",
"spark-daemon.sh",
@@ -350,13 +351,14 @@ try:
"stop-connect-server.sh",
"stop-history-server.sh",
],
- "pyspark.python.lib": ["*.zip"],
- "pyspark.data": ["*.txt", "*.data"],
- "pyspark.licenses": ["*"],
- "pyspark.examples.src.main.python": ["*.py", "*/*.py"],
+ "pyspark.python.lib": ["**/*.zip"],
+ "pyspark.data": ["**/*.txt", "**/*.data"],
+ "pyspark.licenses": ["**/*"],
+ "pyspark.examples.src.main.python": ["**/*.py"],
},
scripts=scripts,
license="Apache-2.0",
+ license_files=["LICENSE", "NOTICE"],
# Don't forget to update python/docs/source/getting_started/install.rst
# if you're updating the versions or dependencies.
install_requires=["py4j>=0.10.9.7,<0.10.9.10"],
diff --git a/python/packaging/client/setup.py b/python/packaging/client/setup.py
index 6c97164937ee..d1754ff64d96 100755
--- a/python/packaging/client/setup.py
+++ b/python/packaging/client/setup.py
@@ -25,7 +25,7 @@
import sys
from setuptools import setup
import os
-from shutil import copyfile, move
+from shutil import copyfile
import glob
from pathlib import Path
@@ -118,13 +118,10 @@ if "SPARK_TESTING" in os.environ:
try:
if in_spark:
- # !!HACK ALTERT!!
- # 1. `setup.py` has to be located with the same directory with the
package.
- # Therefore, we copy the current file, and place it at
`spark/python` directory.
- # After that, we remove it in the end.
- # 2. Here it renames `lib` to `lib.back` so MANIFEST.in does not pick
`py4j` up.
- # We rename it back in the end.
- move("lib", "lib.back")
+ # !!HACK ALERT!!
+ # `setup.py` has to be located with the same directory with the
package.
+ # Therefore, we copy the current file, and place it at `spark/python`
directory.
+ # After that, we remove it in the end.
copyfile("packaging/client/setup.py", "setup.py")
copyfile("packaging/client/setup.cfg", "setup.cfg")
@@ -206,7 +203,11 @@ try:
url="https://github.com/apache/spark/tree/master/python";,
packages=connect_packages + test_packages,
include_package_data=True,
+ package_data={
+ "pyspark": ["**/*.pyi", "**/py.typed", "**/*.json"],
+ },
license="Apache-2.0",
+ license_files=["LICENSE", "NOTICE"],
# Don't forget to update python/docs/source/getting_started/install.rst
# if you're updating the versions or dependencies.
install_requires=[
@@ -232,6 +233,5 @@ try:
)
finally:
if in_spark:
- move("lib.back", "lib")
os.remove("setup.py")
os.remove("setup.cfg")
diff --git a/python/packaging/connect/setup.py
b/python/packaging/connect/setup.py
index a4e83e41d151..5ef47a40fc58 100755
--- a/python/packaging/connect/setup.py
+++ b/python/packaging/connect/setup.py
@@ -25,7 +25,7 @@
import sys
from setuptools import setup
import os
-from shutil import copyfile, copytree, move, rmtree
+from shutil import copyfile, copytree, rmtree
import glob
from pathlib import Path
@@ -57,18 +57,14 @@ in_spark =
os.path.isfile("../core/src/main/scala/org/apache/spark/SparkContext.
try:
if in_spark:
- # !!HACK ALTERT!!
- # 1. `setup.py` has to be located with the same directory with the
package.
- # Therefore, we copy the current file, and place it at
`spark/python` directory.
- # After that, we remove it in the end.
- # 2. Here it renames `pyspark` and `lib` to `pyspark.back` and
`lib.back` so MANIFEST.in
- # does not pick `pyspark` and `py4j` up. We rename it back in the
end.
- move("pyspark", "pyspark.back")
- move("lib", "lib.back")
+ # !!HACK ALERT!!
+ # `setup.py` has to be located with the same directory with the
package.
+ # Therefore, we copy the current file, and place it at `spark/python`
directory.
+ # After that, we remove it in the end.
copyfile("packaging/connect/setup.py", "setup.py")
copyfile("packaging/connect/setup.cfg", "setup.cfg")
copytree("packaging/connect/pyspark_connect", "pyspark_connect")
- copyfile("pyspark.back/version.py", "pyspark_connect/version.py")
+ copyfile("pyspark/version.py", "pyspark_connect/version.py")
try:
exec(open("pyspark_connect/version.py").read())
@@ -114,6 +110,7 @@ try:
packages=connect_packages,
include_package_data=True,
license="Apache-2.0",
+ license_files=["LICENSE", "NOTICE"],
# Don't forget to update python/docs/source/getting_started/install.rst
# if you're updating the versions or dependencies.
install_requires=[
@@ -140,8 +137,6 @@ try:
)
finally:
if in_spark:
- move("pyspark.back", "pyspark")
- move("lib.back", "lib")
os.remove("setup.py")
os.remove("setup.cfg")
rmtree("pyspark_connect")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
