tgravescs commented on issue #26284: [SPARK-29415][Core]Stage Level Sched: Add base ResourceProfile and Request classes URL: https://github.com/apache/spark/pull/26284#issuecomment-548786072 thanks for the comments Holden, I'll try to get a more detailed design and see if I have some test cases that show the usage more clearly. Most of the api is derived from the existing accelerator aware scheduling that is already committed (has global configs). That is already documented: https://github.com/apache/spark/blob/master/docs/configuration.md#custom-resource-scheduling-and-configuration-overview Also see the cluster manager specific ones as it should tell you what is required for that one, like yarn: https://github.com/apache/spark/blob/master/docs/running-on-yarn.md If you don't mind take a look at those docs as its a good test of those and if its not clear I should update to clarify. The discovery script itself isn't used by the cluster manager, its used by the spark executor when it starts up to determine what accelerator addresses the container has made available. the executor registers those back with the driver and the scheduler uses those to assign to specific tasks. Perhaps that is a little confusing because of the ExecutorResourceRequest type, I can definitely document better in that class, if you have ideas on making that more clear in the api let me know.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
