tgravescs commented on issue #25047: [WIP][SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone URL: https://github.com/apache/spark/pull/25047#issuecomment-513898822 So just one general comment, I should have brought this up before. originally I was thinking that a lot of this logic to handle multiple workers would be done by the user (person starting the cluster) or by the discovery script. The user can manually split resources on the node between the workers they are running by passing in the resources file option. They could also do essentially the same thing you are doing here (with locking and resource acquiring) inside of the discovery script. That way if users have different setups they could write something specific to their setup. I can see that if this is the way a lot of people run us providing something there would be more convenient though. But perhaps we should make it pluggable, or at least have that logic be configured off. We had talked about making a class option instead of a discovery script - so a discovery class but didn't see a hard use case for it and thought the script would be easier for users. Again I haven't run standalone in production so will defer to you and others that have. @mengxr If much of this is only being used in standalone mode I think perhaps we should move some of it into the /deploy directory though as well.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
