This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 2f2adcf86c2e [SPARK-57344][INFRA] Ensure tests for `pipelines` module
triggered when sql-related modules are modified
2f2adcf86c2e is described below
commit 2f2adcf86c2edd848fde45d17dbe48f90c063ec3
Author: YangJie <[email protected]>
AuthorDate: Wed Jun 10 16:02:14 2026 +0800
[SPARK-57344][INFRA] Ensure tests for `pipelines` module triggered when
sql-related modules are modified
### What changes were proposed in this pull request?
This PR marks `pipelines` as depending on the `sql` module in Spark's test
module graph, and updates the existing doctest expectations for the affected
module closure.
### Why are the changes needed?
The `pipelines` project depends on Spark SQL, but the test module graph
treated it as independent. As a result, changes in SQL-related modules such as
`sql`, `catalyst`, and `sql-api` did not select `pipelines/test` in the
affected-module CI path.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
```bash
python3 -m py_compile dev/sparktestsupport/modules.py
dev/sparktestsupport/utils.py
PYTHONPATH=dev python3 -m doctest dev/sparktestsupport/utils.py
git diff --check
```
Also verified locally that changes in `core`, `sql-api`, `catalyst`, `sql`,
and `sql/pipelines` include the `pipelines` module in the affected-module
selection.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
Closes #56405 from LuciferYang/fix-pipelines-module-deps.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
(cherry picked from commit 6d4b71ef52b9b08bcfd56dbce1ac913a3c8bf2e7)
Signed-off-by: yangjie01 <[email protected]>
---
dev/sparktestsupport/modules.py | 2 +-
dev/sparktestsupport/utils.py | 14 +++++++-------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index ef5859bbfe43..9853e4dfe7a6 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -449,7 +449,7 @@ mllib = Module(
pipelines = Module(
name="pipelines",
- dependencies=[],
+ dependencies=[sql],
source_file_regexes=["sql/pipelines"],
sbt_test_goals=[
"pipelines/test",
diff --git a/dev/sparktestsupport/utils.py b/dev/sparktestsupport/utils.py
index fe795303e63f..fff418bdb348 100755
--- a/dev/sparktestsupport/utils.py
+++ b/dev/sparktestsupport/utils.py
@@ -104,8 +104,8 @@ def determine_modules_to_test(changed_modules,
deduplicated=True):
>>> sorted([x.name for x in determine_modules_to_test([modules.sql])])
... # doctest: +NORMALIZE_WHITESPACE
['avro', 'connect', 'docker-integration-tests', 'examples', 'hive',
'hive-thriftserver',
- 'mllib', 'protobuf', 'pyspark-connect', 'pyspark-ml',
'pyspark-ml-connect', 'pyspark-mllib',
- 'pyspark-pandas', 'pyspark-pandas-connect', 'pyspark-pandas-slow',
+ 'mllib', 'pipelines', 'protobuf', 'pyspark-connect', 'pyspark-ml',
'pyspark-ml-connect',
+ 'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-connect',
'pyspark-pandas-slow',
'pyspark-pandas-slow-connect', 'pyspark-pipelines', 'pyspark-sql',
'pyspark-structured-streaming', 'pyspark-structured-streaming-connect',
'pyspark-testing', 'repl', 'sparkr', 'sql', 'sql-kafka-0-10']
@@ -113,8 +113,8 @@ def determine_modules_to_test(changed_modules,
deduplicated=True):
... [modules.sparkr, modules.sql], deduplicated=False)])
... # doctest: +NORMALIZE_WHITESPACE
['avro', 'connect', 'docker-integration-tests', 'examples', 'hive',
'hive-thriftserver',
- 'mllib', 'protobuf', 'pyspark-connect', 'pyspark-ml',
'pyspark-ml-connect', 'pyspark-mllib',
- 'pyspark-pandas', 'pyspark-pandas-connect', 'pyspark-pandas-slow',
+ 'mllib', 'pipelines', 'protobuf', 'pyspark-connect', 'pyspark-ml',
'pyspark-ml-connect',
+ 'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-connect',
'pyspark-pandas-slow',
'pyspark-pandas-slow-connect', 'pyspark-pipelines', 'pyspark-sql',
'pyspark-structured-streaming', 'pyspark-structured-streaming-connect',
'pyspark-testing', 'repl', 'sparkr', 'sql', 'sql-kafka-0-10']
@@ -122,9 +122,9 @@ def determine_modules_to_test(changed_modules,
deduplicated=True):
... [modules.sql, modules.core], deduplicated=False)])
... # doctest: +NORMALIZE_WHITESPACE
['avro', 'catalyst', 'connect', 'core', 'docker-integration-tests',
'examples', 'graphx',
- 'hive', 'hive-thriftserver', 'mllib', 'mllib-local', 'protobuf',
'pyspark-connect',
- 'pyspark-core', 'pyspark-install', 'pyspark-ml', 'pyspark-ml-connect',
'pyspark-mllib',
- 'pyspark-pandas', 'pyspark-pandas-connect', 'pyspark-pandas-slow',
+ 'hive', 'hive-thriftserver', 'mllib', 'mllib-local', 'pipelines',
'protobuf',
+ 'pyspark-connect', 'pyspark-core', 'pyspark-install', 'pyspark-ml',
'pyspark-ml-connect',
+ 'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-connect',
'pyspark-pandas-slow',
'pyspark-pandas-slow-connect', 'pyspark-pipelines', 'pyspark-resource',
'pyspark-sql',
'pyspark-streaming', 'pyspark-structured-streaming',
'pyspark-structured-streaming-connect',
'pyspark-testing', 'repl', 'root', 'sparkr', 'sql', 'sql-kafka-0-10',
'streaming',
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
