This is an automated email from the ASF dual-hosted git repository.
nchammas pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c535eb65a893 [SPARK-59577][BUILD] Add shared PySpark base test module
c535eb65a893 is described below
commit c535eb65a8931b67e277d3fa9e77c9717649e674
Author: Nicholas Chammas <[email protected]>
AuthorDate: Mon Oct 26 09:58:39 2026 +0200
[SPARK-59577][BUILD] Add shared PySpark base test module
### What changes were proposed in this pull request?
Add a new `pyspark-base` test module to capture test triggers that should
be common to all PySpark modules, and update the PySpark module dependencies
accordingly.
Add a new doctest to ensure that all PySpark modules ultimately depend on
`pyspark-base`.
### Why are the changes needed?
Changes to those Dockerfiles impact the containers used to run PySpark
tests. They are built as part of `build_infra_images_cache.yml`. When the test
environment changes, we need to rerun the tests to ensure they still pass.
Right now we are doing this by allowing the test selection in
`sparktestsupport` to fall through to `root` and build absolutely everything.
This is extremely wasteful. Instead, we just need to build the PySpark modules.
Since they currently do not have a common base (`pyspark-periodic` and
`pyspark-logger` do not depend on `pyspark-core`), this PR adds a new base so
that all PySpark modules can share this test trigger.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI here.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #58860 from nchammas/pyspark-core-docker-test-paths.
Authored-by: Nicholas Chammas <[email protected]>
Signed-off-by: Nicholas Chammas <[email protected]>
---
dev/sparktestsupport/modules.py | 48 ++++++++++++++++++++++++++++++++++++++---
1 file changed, 45 insertions(+), 3 deletions(-)
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 17f2252e10e2..24a5348e6c9e 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -596,9 +596,19 @@ examples = Module(
],
)
+# Shared Python test environment.
+pyspark_base = Module(
+ name="pyspark-base",
+ dependencies=[],
+ source_file_regexes=[
+ "dev/spark-test-image/python-.*/",
+ "pyproject.toml",
+ ],
+)
+
pyspark_core = Module(
name="pyspark-core",
- dependencies=[core],
+ dependencies=[core, pyspark_base],
source_file_regexes=[
"python/(?!pyspark/(ml|mllib|sql|streaming|pandas|resource|testing|inprocess))"
],
@@ -948,7 +958,7 @@ pyspark_ml = Module(
pyspark_periodic = Module(
name="pyspark-periodic",
- dependencies=[],
+ dependencies=[pyspark_base],
source_file_regexes=[
# This module contains tests that are not sensitive to pyspark code
changes.
# We run these on scheduled CIs and pre-merge CIs, not post-merge CIs.
@@ -1746,7 +1756,7 @@ pyspark_errors = Module(
pyspark_logger = Module(
name="pyspark-logger",
- dependencies=[],
+ dependencies=[pyspark_base],
source_file_regexes=["python/pyspark/logger"],
python_test_goals=[
# doctests
@@ -1874,6 +1884,38 @@ root = Module(
)
+def transitive_dependencies(module: Module) -> set[Module]:
+ """
+ Return all modules that `module` depends on, directly or transitively.
+ The module itself is not included.
+
+ `utils-java` is an indirect dependency of `unsafe`:
+
+ >>> sorted(dependency.name for dependency in
transitive_dependencies(unsafe))
+ ['config', 'tags', 'utils', 'utils-java']
+
+ Every `pyspark-*` module must depend on `pyspark-base`:
+
+ >>> [
+ ... module.name
+ ... for module in all_modules
+ ... if module.name.startswith("pyspark-")
+ ... and module is not pyspark_base
+ ... and pyspark_base not in transitive_dependencies(module)
+ ... ]
+ []
+ """
+ seen = set()
+ pending = list(module.dependencies)
+ while pending:
+ dependency = pending.pop()
+ if dependency in seen or dependency is module:
+ continue
+ seen.add(dependency)
+ pending.extend(dependency.dependencies)
+ return seen
+
+
def _test():
import doctest
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
