lu-wang-dl commented on code in PR #42308:
URL: https://github.com/apache/spark/pull/42308#discussion_r1284718685
##########
python/pyspark/ml/deepspeed/deepspeed_distributor.py:
##########
@@ -97,6 +97,13 @@ def __init__(
_ssl_conf=DeepspeedTorchDistributor._DEEPSPEED_SSL_CONF,
)
self.cleanup_deepspeed_conf = False
+ if not useGpu:
+ # Note: this is because CPU is not *as* well supported by
+ # deepspeed itself. Once Deepspeed + Intel and others better
+ # support this feature, it will be safe to remove this error
+ # because the command we generate to run deepspeed should work
+ # fine with CPUs.
+ raise RuntimeError("CPUs aren't supported at this time.")
Review Comment:
Could we make the error message more clear? Or can we just remove the
`useGpu` param and showing error message when not getting GPUs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
