This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5425d453d244 [SPARK-50394][PYTHON][INFRA][FOLLOW-UP] Reduce
parallelism further in Pure Python library builds
5425d453d244 is described below
commit 5425d453d2442079377ce0fbcf667f70d35e256e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Nov 27 09:42:25 2024 +0900
[SPARK-50394][PYTHON][INFRA][FOLLOW-UP] Reduce parallelism further in Pure
Python library builds
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/48932 that
reduces parallelism further
### Why are the changes needed?
In order to make the tests more robust:
https://github.com/apache/spark/actions/workflows/build_python_connect.yml
It fails because of OOM.
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Will monitor the build.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48977 from HyukjinKwon/SPARK-50394-followup2.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/build_python_connect.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/build_python_connect.yml
b/.github/workflows/build_python_connect.yml
index d57a0c2b9162..471ad31279da 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -93,7 +93,7 @@ jobs:
# Several tests related to catalog requires to run them
sequencially, e.g., writing a table in a listener.
./python/run-tests --parallelism=1 --python-executables=python3
--modules pyspark-connect,pyspark-ml-connect
# None of tests are dependent on each other in Pandas API on Spark
so run them in parallel
- ./python/run-tests --parallelism=2 --python-executables=python3
--modules
pyspark-pandas-connect-part0,pyspark-pandas-connect-part1,pyspark-pandas-connect-part2,pyspark-pandas-connect-part3
+ ./python/run-tests --parallelism=1 --python-executables=python3
--modules
pyspark-pandas-connect-part0,pyspark-pandas-connect-part1,pyspark-pandas-connect-part2,pyspark-pandas-connect-part3
# Stop Spark Connect server.
./sbin/stop-connect-server.sh
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
