This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0824934103b1 [SPARK-54643][SDP][PYTHON][TESTS] Run missing pyspark
pipelines tests in CI
0824934103b1 is described below
commit 0824934103b17c3aad50262ce929a54ad4c0d0f9
Author: Sandy Ryza <[email protected]>
AuthorDate: Tue Dec 9 07:59:02 2025 -0800
[SPARK-54643][SDP][PYTHON][TESTS] Run missing pyspark pipelines tests in CI
### What changes were proposed in this pull request?
Adds two pyspark pipelines tests to CI, and fixes one of them.
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Ran testes locally.
### Was this patch authored or co-authored using generative AI tooling?
Closes #53393 from sryza/pipelines-test-module.
Authored-by: Sandy Ryza <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/sparktestsupport/modules.py | 2 ++
python/pyspark/pipelines/tests/test_spark_connect.py | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 306a3b69223f..ee2a953a7402 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -1566,11 +1566,13 @@ pyspark_pipelines = Module(
dependencies=[pyspark_core, pyspark_sql, pyspark_connect],
source_file_regexes=["python/pyspark/pipelines"],
python_test_goals=[
+ "pyspark.pipelines.tests.test_add_pipeline_analysis_context",
"pyspark.pipelines.tests.test_block_session_mutations",
"pyspark.pipelines.tests.test_cli",
"pyspark.pipelines.tests.test_decorators",
"pyspark.pipelines.tests.test_graph_element_registry",
"pyspark.pipelines.tests.test_init_cli",
+ "pyspark.pipelines.tests.test_spark_connect",
],
)
diff --git a/python/pyspark/pipelines/tests/test_spark_connect.py
b/python/pyspark/pipelines/tests/test_spark_connect.py
index 0b54c0906f9a..3027de46864f 100644
--- a/python/pyspark/pipelines/tests/test_spark_connect.py
+++ b/python/pyspark/pipelines/tests/test_spark_connect.py
@@ -58,7 +58,7 @@ class SparkConnectPipelinesTest(ReusedConnectTestCase):
refresh=None,
full_refresh_all=False,
dry=True,
- storage="storage_path",
+ storage="file:///tmp/storage_path",
)
handle_pipeline_events(result_iter)
@@ -80,7 +80,7 @@ class SparkConnectPipelinesTest(ReusedConnectTestCase):
refresh=None,
full_refresh_all=False,
dry=True,
- storage="storage_path",
+ storage="file:///tmp/storage_path",
)
with self.assertRaises(AnalysisException) as context:
handle_pipeline_events(result_iter)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
