Ngone51 opened a new pull request #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841 ## What changes were proposed in this pull request? To support GPU-aware scheduling in Standalone (cluster mode), Worker should have ability to setup resources(e.g. GPU/FPGA) when it starts up. Similar as driver/executor do, Worker has two ways(resourceFile & resourceDiscoveryScript) to setup resources when it starts up. And, resourceFile takes first if both are configured by user. Like what we always do for configuring Working, we could config resources by conf/spark-env.sh(using `SPARK_WORKER_RESOURCE_FILE` & `SPARK_WORKER_RESOURCE_DISCOVERY_SCRIPT`) or sbin/start-slave.sh(using `--resource-file` & `--resource-script`). Note that, configure format should be, e.g. `gpu:/path1/to1/gpu-script.sh,fpga:/path2/to2/fpga-script.sh`, when using discovery script to setup resources. ## How was this patch tested? Tested manually in Standalone locally: - Worker could start up normally when no resources are configured - Worker should fail to start up when exception threw during setup resources(e.g. unknown directory, parse fail) - Worker could setup resources from resource file - Worker could setup resources from discovery scripts - Worker should setup resources from resource file when both are configure.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
