This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 175b5e9d7f80 [MINOR][BUILD] Skip `deepspeed` in requirements on MacOS
175b5e9d7f80 is described below
commit 175b5e9d7f80167d835fd229efa461a0012228c2
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Thu Aug 10 12:12:11 2023 +0900
[MINOR][BUILD] Skip `deepspeed` in requirements on MacOS
### What changes were proposed in this pull request?
`deepspeed` is not supported on MacOS
### Why are the changes needed?
to fix this on MacOS
```
pip install -U -r dev/requirements.txt
...
Collecting deepspeed (from -r dev/requirements.txt (line 69))
Using cached deepspeed-0.10.0.tar.gz (836 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File
"/private/var/folders/l_/b6xgqlvx0895dljz46x9nl780000gp/T/pip-install-zd43o1nk/deepspeed_47bbb5784bc942e6bdf0f5ec24e9f939/setup.py",
line 37, in <module>
from op_builder.all_ops import ALL_OPS
File
"/private/var/folders/l_/b6xgqlvx0895dljz46x9nl780000gp/T/pip-install-zd43o1nk/deepspeed_47bbb5784bc942e6bdf0f5ec24e9f939/op_builder/all_ops.py",
line 29, in <module>
builder = get_accelerator().create_op_builder(member_name)
File
"/private/var/folders/l_/b6xgqlvx0895dljz46x9nl780000gp/T/pip-install-zd43o1nk/deepspeed_47bbb5784bc942e6bdf0f5ec24e9f939/accelerator/mps_accelerator.py",
line 211, in create_op_builder
builder_class = self.get_op_builder(op_name)
File
"/private/var/folders/l_/b6xgqlvx0895dljz46x9nl780000gp/T/pip-install-zd43o1nk/deepspeed_47bbb5784bc942e6bdf0f5ec24e9f939/accelerator/mps_accelerator.py",
line 218, in get_op_builder
from deepspeed.ops.op_builder.cpu import NotImplementedBuilder
File
"/private/var/folders/l_/b6xgqlvx0895dljz46x9nl780000gp/T/pip-install-zd43o1nk/deepspeed_47bbb5784bc942e6bdf0f5ec24e9f939/deepspeed/__init__.py",
line 21, in <module>
from . import ops
File
"/private/var/folders/l_/b6xgqlvx0895dljz46x9nl780000gp/T/pip-install-zd43o1nk/deepspeed_47bbb5784bc942e6bdf0f5ec24e9f939/deepspeed/ops/__init__.py",
line 6, in <module>
from . import adam
File
"/private/var/folders/l_/b6xgqlvx0895dljz46x9nl780000gp/T/pip-install-zd43o1nk/deepspeed_47bbb5784bc942e6bdf0f5ec24e9f939/deepspeed/ops/adam/__init__.py",
line 6, in <module>
from .cpu_adam import DeepSpeedCPUAdam
File
"/private/var/folders/l_/b6xgqlvx0895dljz46x9nl780000gp/T/pip-install-zd43o1nk/deepspeed_47bbb5784bc942e6bdf0f5ec24e9f939/deepspeed/ops/adam/cpu_adam.py",
line 7, in <module>
from cpuinfo import get_cpu_info
ModuleNotFoundError: No module named 'cpuinfo'
[end of output]
note: This error originates from a subprocess, and is likely not a
problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
manually check
Closes #42411 from zhengruifeng/install_deepspeed_on_linux.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 5c94565e75cc32e552414f27da8082591121a86d)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/requirements.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev/requirements.txt b/dev/requirements.txt
index 0749af75aa4b..e3c3cae59d05 100644
--- a/dev/requirements.txt
+++ b/dev/requirements.txt
@@ -67,5 +67,5 @@ torchvision
torcheval
# DeepspeedTorchDistributor dependencies
-deepspeed
+deepspeed; sys_platform != 'darwin'
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
