tgravescs commented on a change in pull request #24615: [SPARK-27488][CORE]
Driver interface to support GPU resources
URL: https://github.com/apache/spark/pull/24615#discussion_r286183567
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -360,6 +363,48 @@ class SparkContext(config: SparkConf) extends Logging {
Utils.setLogLevel(org.apache.log4j.Level.toLevel(upperCased))
}
+ /**
+ * Checks to see if any resources (GPU/FPGA/etc) are available to the driver
by looking
+ * at and processing the spark.driver.resource.resourceName.addresses and
+ * spark.driver.resource.resourceName.discoveryScript configs. The configs
have to be
+ * present when the driver starts, setting them after startup does not work.
+ *
+ * If any resource addresses configs were specified then assume all
resources will be specified
+ * in that way. Otherwise use the discovery scripts to find the resources.
Users should
+ * not really be setting the addresses config directly and should not be
mixing methods
+ * for different types of resources since the addresses config is meant for
Standalone mode
+ * and other cluster managers should use the discovery scripts.
+ */
+ private def setupDriverResources(): Unit = {
+ // Only call getAllWithPrefix once and filter on those since there could
be a lot of spark
+ // configs.
+ val allDriverResourceConfs =
_conf.getAllWithPrefix(SPARK_DRIVER_RESOURCE_PREFIX)
+ val resourcesWithAddrsInConfs =
+ SparkConf.getConfigsWithSuffix(allDriverResourceConfs,
SPARK_RESOURCE_ADDRESSES_SUFFIX)
+
+ _resources = if (resourcesWithAddrsInConfs.nonEmpty) {
Review comment:
in what case are you thinking? The address config is really an internal
config meant for standalone mode.
The only case I can think of is if a user requests a resource the standalone
mode worker handles but then asks for another resource that it doesn't and thus
needs discoveryScript. But that means that they are on their own for handling
isolation and assignment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
