jiangxb1987 opened a new pull request #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374 ## What changes were proposed in this pull request? This PR adds support to schedule tasks with GPU resource requirements on executors with available GPU resources. A few major features introduced in this PR: 1. Represent resources by `ResourceInformation`; 2. Introduce new config `spark.task.resource.gpu.count` to specify number of GPUs required by each task; 3. Introduce new configs to specify number of GPUs assigned to each executor/driver; 4. Support two ways to pass in GPU resources to an executor: * Specify GPU devices by argument `--gpu-devices` passed to `CoarseGrainedExecutorBackend`; * Provide GPU devices auto discovery script through `spark.executor.resource.gpu.discoveryScript`; 5. Provide `TaskContext.resources()` method so tasks can access available device indices allocated to them. ## How was this patch tested? * Added new end-to-end test cases in `SparkContextSuite`; * Added new test case in `CoarseGrainedSchedulerBackendSuite`; * Added new test case in `TaskSchedulerImplSuite`; * Added new test suite `ResourceDiscovererSuite`; * Updated existing tests. **Please note this PR is split out from the branch https://github.com/tgravescs/spark/tree/gpu-sched, credits shall goes to Thomas Graves since he contributed most of the design and code.**
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
