This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 4e0b88a [SPARK-59096] Fully vendor pyspark.pandas with a drift check
and keep tests out of the wheel
4e0b88a is described below
commit 4e0b88ad5242bb498d00d09e174505846d2b9cac
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Aug 29 15:25:20 2026 +0900
[SPARK-59096] Fully vendor pyspark.pandas with a drift check and keep tests
out of the wheel
### What changes were proposed in this pull request?
Make the entire `pyspark.pandas` (pandas-on-Spark) package a
fully-vendored, drift-checked mirror of Apache Spark, and keep tests out of the
packaged wheel.
- **Vendored-drift CI** (`scripts/check_vendored_upstream.sh`): replace the
per-subdirectory / per-file `pyspark/pandas` entries with the whole `pandas`
tree, compared byte-for-byte against the `v4.2.0` tag. `check_one` now takes an
optional extra `--exclude`, and `pandas` is checked with `--exclude=tests`
because we intentionally do not ship `pandas/tests`. Any hand-edit to a file
under `pyspark/pandas` now fails CI. (Our `pyspark/pandas` is already
byte-identical to upstream, so no [...]
- **Packaging** (`pyproject.toml`): add a maturin `exclude` for
`pyspark/**/tests/**` and `pyspark/**/test_*.py`, so the vendored upstream test
modules (`pyspark/pipelines/tests`, `pyspark/testing/tests`) stay out of the
built wheel/sdist.
### Why are the changes needed?
`pyspark.pandas` is vendored verbatim from Apache Spark; it must never be
locally edited (edits would silently diverge from upstream and break the
"drop-in" guarantee). Enforcing the whole tree in the drift gate makes that a
hard CI failure instead of a convention. Excluding tests keeps the shipped
package lean and free of upstream test suites end users don't need.
### How was this patch tested?
- `diff -r --exclude=tests --exclude=__pycache__ python/pyspark/pandas
<v4.2.0>/python/pyspark/pandas` → no differences (the tree matches upstream).
- `bash -n scripts/check_vendored_upstream.sh` passes; the new whole-tree
`pandas` check with the `tests` exclude reports OK against a local v4.2.0
checkout.
- `maturin sdist` → 0 Python test files packaged (`pyspark/**/tests`,
`test_*.py`), pandas modules intact.
This pull request and its description were written by Isaac.
Closes #82 from HyukjinKwon/vendor-pandas-drift.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
pyproject.toml | 9 ++++++
scripts/check_vendored_upstream.sh | 58 +++++++++++++-------------------------
2 files changed, 28 insertions(+), 39 deletions(-)
diff --git a/pyproject.toml b/pyproject.toml
index 0ec2245..a32c09a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -63,3 +63,12 @@ module-name = "pyspark._pyspark"
manifest-path = "crates/pyspark-rs/Cargo.toml"
# Use abi3 for a stable ABI compatible across Python 3.9+.
features = ["pyo3/abi3-py39", "pyo3/extension-module"]
+# Keep tests out of the built wheel/sdist: our own offline suite lives outside
the package
+# (python/tests), but the vendored pyspark tree carries upstream test modules
+# (e.g. pyspark/pipelines/tests, pyspark/testing/tests) that end users don't
need.
+exclude = [
+ "python/pyspark/**/tests/**",
+ "python/pyspark/**/test_*.py",
+ "pyspark/**/tests/**",
+ "pyspark/**/test_*.py",
+]
diff --git a/scripts/check_vendored_upstream.sh
b/scripts/check_vendored_upstream.sh
index 634d23e..e96a84e 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -32,13 +32,10 @@ SPARK_REPO="https://github.com/apache/spark.git";
# Directories under python/pyspark vendored verbatim from upstream (whole tree
byte-identical).
VENDORED_DIRS=(
"cloudpickle"
- "pandas/data_type_ops"
- "pandas/indexes"
- "pandas/missing"
- "pandas/plot"
- "pandas/spark"
- "pandas/typedef"
- "pandas/usage_logging"
+ # The entire pandas-on-Spark package is vendored byte-for-byte from upstream
(only its
+ # tests/ subdir is intentionally not shipped; excluded in check_one). Do NOT
hand-edit any
+ # file under pyspark/pandas -- re-vendor from Apache instead.
+ "pandas"
"mllib/linalg"
"sql/worker"
"sql/plot"
@@ -123,34 +120,8 @@ VENDORED_FILES=(
"mllib/tree.py"
"mllib/util.py"
- # Pandas: core (excluding adapted group_ops, functions, and Connect-specific
modules)
- "pandas/__init__.py"
- "pandas/_typing.py"
- "pandas/accessors.py"
- "pandas/base.py"
- "pandas/categorical.py"
- "pandas/config.py"
- "pandas/correlation.py"
- "pandas/datetimes.py"
- "pandas/exceptions.py"
- "pandas/extensions.py"
- "pandas/frame.py"
- "pandas/generic.py"
- "pandas/groupby.py"
- "pandas/indexing.py"
- "pandas/internal.py"
- "pandas/mlflow.py"
- "pandas/namespace.py"
- "pandas/numpy_compat.py"
- "pandas/resample.py"
- "pandas/series.py"
- "pandas/sql_formatter.py"
- "pandas/sql_processor.py"
- "pandas/strings.py"
- "pandas/supported_api_gen.py"
- "pandas/testing.py"
- "pandas/utils.py"
- "pandas/window.py"
+ # Pandas: the whole pyspark/pandas tree is vendored as a VENDORED_DIRS entry
above (every
+ # file, tests/ excluded), so no individual pandas/*.py entries are listed
here.
# SQL: pandas (excluding adapted group_ops, functions)
"sql/pandas/__init__.py"
@@ -203,25 +174,34 @@ git clone --quiet --depth 1 --branch "${SPARK_TAG}"
--filter=blob:none --sparse
status=0
check_one() {
- local p="$1" flag="$2"
+ local p="$1" flag="$2" extra_exclude="${3:-}"
local ours="${REPO_ROOT}/python/pyspark/${p}"
local theirs="${WORK}/spark/python/pyspark/${p}"
+ # Always ignore __pycache__; a caller may add one more --exclude (e.g.
`tests` for the
+ # whole pandas tree, which upstream ships but we intentionally do not
vendor).
+ local excludes=(--exclude=__pycache__)
+ [[ -n "${extra_exclude}" ]] && excludes+=(--exclude="${extra_exclude}")
if [[ ! -e "${theirs}" ]]; then
echo "ERROR: upstream path python/pyspark/${p} not found at ${SPARK_TAG}"
status=1
return
fi
- if diff ${flag} --exclude=__pycache__ "${ours}" "${theirs}" >/dev/null; then
+ if diff ${flag} "${excludes[@]}" "${ours}" "${theirs}" >/dev/null; then
echo "OK python/pyspark/${p} matches Apache Spark ${SPARK_TAG}"
else
echo "DRIFT python/pyspark/${p} differs from Apache Spark ${SPARK_TAG}:"
- diff ${flag} --exclude=__pycache__ "${ours}" "${theirs}" || true
+ diff ${flag} "${excludes[@]}" "${ours}" "${theirs}" || true
status=1
fi
}
for p in "${VENDORED_DIRS[@]}"; do
- check_one "${p}" "-r"
+ # The pandas-on-Spark tree is vendored in full EXCEPT its tests/ (which we
do not ship).
+ if [[ "${p}" == "pandas" ]]; then
+ check_one "${p}" "-r" "tests"
+ else
+ check_one "${p}" "-r"
+ fi
done
for p in "${VENDORED_FILES[@]}"; do
check_one "${p}" ""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
