This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 48367e65dff2 [SPARK-57981][PYTHON][TEST] Move packaging test with
test-install
48367e65dff2 is described below
commit 48367e65dff2958e68603e6293484c22eba108b8
Author: Tian Gao <[email protected]>
AuthorDate: Tue Jul 7 11:57:05 2026 -0700
[SPARK-57981][PYTHON][TEST] Move packaging test with test-install
### What changes were proposed in this pull request?
1. We have a python packaging test which used to run with
`pyspark-pipelines`. We move it to run with `pyspark-install`.
2. A few month ago, the packaging test executed with `pyspark-errors` so
there's some legacy code to trigger `pyspark-errors` for any changes in
`python` - that was removed.
3. We don't trigger `pyspark` tests anymore for `pyspark-install`.
### Why are the changes needed?
Python packaging test takes about 5 minutes. It's a low risk test which
does not really test any functionality. Running it for every commit is not
necessary. With this change, this test will run pre-merge for each PR, and in
scheduled CI if it's explicitly specified (`build_coverage` for now). This will
save us some minutes for workers.
Also, it's very natural to run packaging tests with installation tests -
running it with either pipeline or error is just very arbitrary.
More importantly, the current CI setup will run all pyspark tests if any of
the pyspark test needs to run. With the existing dependency, all pyspark tests
will run if anything changes in `python/`, including `python/docs` or
`python/packaging`, which is unnecessary.
After this change, we can skip post-merge pyspark tests if the change is
not in python/pyspark, which will save us some minutes too.
### Does this PR introduce _any_ user-facing change?
No, this is CI only.
### How was this patch tested?
CI should confirm that packaging test does not run with `pyspark-pipelines`
anymore.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57060 from gaogaotiantian/move-packaging-test.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 5057e0d56104adeafa648e59e18c736d9c326ea8)
Signed-off-by: Tian Gao <[email protected]>
---
.github/workflows/build_and_test.yml | 4 ++--
dev/sparktestsupport/modules.py | 13 ++++++-------
dev/sparktestsupport/utils.py | 4 ++--
3 files changed, 10 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index d15ceef88a3f..75152cb5e063 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -97,7 +97,7 @@ jobs:
id: set-outputs
run: |
if [ -z "${{ inputs.jobs }}" ]; then
- pyspark_modules=`cd dev && python -c "import
sparktestsupport.modules as m; print(','.join(m.name for m in m.all_modules if
m.name.startswith('pyspark') and not m.name.startswith('pyspark-pandas')))"`
+ pyspark_modules=`cd dev && python -c "import
sparktestsupport.modules as m; print(','.join(m.name for m in m.all_modules if
m.name.startswith('pyspark') and not m.name.startswith('pyspark-pandas') and
not m.name == 'pyspark-install'))"`
pyspark_pandas_modules=`cd dev && python -c "import
sparktestsupport.modules as m; print(','.join(m.name for m in m.all_modules if
m.name.startswith('pyspark-pandas')))"`
pyspark=`./dev/is-changed.py -m $pyspark_modules`
pandas=`./dev/is-changed.py -m $pyspark_pandas_modules`
@@ -792,7 +792,7 @@ jobs:
export SKIP_SCALA_BUILD=true
echo "Reusing precompiled artifact, skipping local SBT build."
fi
- if [[ "$MODULES_TO_TEST" == *"pyspark-pipelines"* ]]; then
+ if [[ "$MODULES_TO_TEST" == *"pyspark-install"* ]]; then
export SKIP_PACKAGING=false
echo "Python Packaging Tests Enabled!"
fi
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index f929a5d2c6b3..8f1395341884 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -817,8 +817,12 @@ pyspark_ml = Module(
pyspark_install = Module(
name="pyspark-install",
- dependencies=[core],
+ dependencies=[],
source_file_regexes=[
+ # Python package tests will be triggered with this module
+ # Any changes in python/ should trigger this module
+ # This module won't be executed for post-commit CIs so it's cheap
+ "python/",
"python/pyspark/install.py",
"python/pyspark/tests/test_install_spark.py",
],
@@ -1559,13 +1563,8 @@ pyspark_pandas_slow_connect = Module(
pyspark_errors = Module(
name="pyspark-errors",
- dependencies=[],
+ dependencies=[pyspark_core],
source_file_regexes=[
- # SPARK-44544: Force the execution of pyspark_errors when there are
any changes
- # in PySpark, since the Python Packaging Tests is only enabled within
this module.
- # This module is the smallest Python test module, it contains only 1
test file
- # and normally takes < 2 seconds, so the additional cost is small.
- "python/",
"python/pyspark/errors",
],
python_test_goals=[
diff --git a/dev/sparktestsupport/utils.py b/dev/sparktestsupport/utils.py
index fff418bdb348..029b1627bd0b 100755
--- a/dev/sparktestsupport/utils.py
+++ b/dev/sparktestsupport/utils.py
@@ -36,7 +36,7 @@ def determine_modules_for_files(filenames):
file to belong to the 'root' module. `.github` directory is counted only
in GitHub Actions.
>>> sorted(x.name for x in
determine_modules_for_files(["python/pyspark/a.py", "sql/core/foo"]))
- ['pyspark-core', 'pyspark-errors', 'sql']
+ ['pyspark-core', 'pyspark-install', 'sql']
>>> [x.name for x in
determine_modules_for_files(["file_not_matched_by_any_subproject"])]
['root']
"""
@@ -123,7 +123,7 @@ def determine_modules_to_test(changed_modules,
deduplicated=True):
... # doctest: +NORMALIZE_WHITESPACE
['avro', 'catalyst', 'connect', 'core', 'docker-integration-tests',
'examples', 'graphx',
'hive', 'hive-thriftserver', 'mllib', 'mllib-local', 'pipelines',
'protobuf',
- 'pyspark-connect', 'pyspark-core', 'pyspark-install', 'pyspark-ml',
'pyspark-ml-connect',
+ 'pyspark-connect', 'pyspark-core', 'pyspark-errors', 'pyspark-ml',
'pyspark-ml-connect',
'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-connect',
'pyspark-pandas-slow',
'pyspark-pandas-slow-connect', 'pyspark-pipelines', 'pyspark-resource',
'pyspark-sql',
'pyspark-streaming', 'pyspark-structured-streaming',
'pyspark-structured-streaming-connect',
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
