rithwik-db commented on code in PR #42308:
URL: https://github.com/apache/spark/pull/42308#discussion_r1283523318
##########
python/pyspark/ml/deepspeed/deepspeed_distributor.py:
##########
@@ -97,6 +98,8 @@ def __init__(
_ssl_conf=DeepspeedTorchDistributor._DEEPSPEED_SSL_CONF,
)
self.cleanup_deepspeed_conf = False
+ if not useGpu:
+ warnings.warn(message="Deepspeed usage with CPUs is not very well
supported at this time.")
Review Comment:
But we haven't successfully tested it in our use case, right? If they do run
into errors, they will probably message oncall and we don't have a plan to
approach such fixes. I will let @lu-wang-dl make the final decision. Should we
raise a warning or an error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
