This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6d1be3a389e [SPARK-44667][INFRA][FOLLOWUP] Uninstall `deepspeed`
libraries for non-ML jobs
6d1be3a389e is described below
commit 6d1be3a389e407cb1422fa50e4b29270415d8fe5
Author: Ruifeng Zheng <ruife...@apache.org>
AuthorDate: Mon Sep 4 09:45:49 2023 +0900
[SPARK-44667][INFRA][FOLLOWUP] Uninstall `deepspeed` libraries for non-ML
jobs
### What changes were proposed in this pull request?
Uninstall `deepspeed` libraries for non-ML jobs
### Why are the changes needed?
after https://github.com/apache/spark/pull/42334, `deepspeed` was
introduced as a dependency, which is not needed for non-ML jobs
### Does this PR introduce _any_ user-facing change?
no, dev-only
### How was this patch tested?
CI
### Was this patch authored or co-authored using generative AI tooling?
NO
Closes #42768 from zhengruifeng/uninstall_deepspeed.
Authored-by: Ruifeng Zheng <ruife...@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
.github/workflows/build_and_test.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index beb5a7772b7..f0bd65bcf41 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -425,7 +425,7 @@ jobs:
run: |
if [[ "$MODULES_TO_TEST" != *"pyspark-ml"* ]] && [[ "$BRANCH" !=
"branch-3.5" ]]; then
# uninstall libraries dedicated for ML testing
- python3.9 -m pip uninstall -y torch torchvision torcheval torchtnt
tensorboard mlflow
+ python3.9 -m pip uninstall -y torch torchvision torcheval torchtnt
tensorboard mlflow deepspeed
fi
if [ -f ./dev/free_disk_space_container ]; then
./dev/free_disk_space_container
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
