revans2 commented on issue #27410: [SPARK-30689][CORE][YARN][test-java11] Add resource discovery plugin api to support YARN versions with resource scheduling URL: https://github.com/apache/spark/pull/27410#issuecomment-580769594 Looking at the code I am a little confused about how to use this. Especially if there are multiple resources involved each might want custom code and be vendor specific. As such I really would like to see one of two things happen. Either have the core of `ResourceDiscoveryScriptPlugin.discoverResource` be a static method that other plugins can call so they don't have to rewrite the discovery script logic, or change the config/API so that `discoverResource` returns an `Optional[ResourceInformation]`. Then have a list of plugins with the `ResourceDiscoveryScriptPlugin` being a final one that you cannot turn off, and will always return a resource. This way one vendor can write a GPU specific plugin, and another vendor can write an FPGA specific one, and there would be no contention or overlap between them, and none of them would have to know about the other or the scripting.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
