tgravescs commented on a change in pull request #24406: [SPARK-27024] Executor
interface for cluster managers to support GPU and other resources
URL: https://github.com/apache/spark/pull/24406#discussion_r281652926
##########
File path:
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
##########
@@ -44,9 +52,12 @@ private[spark] class CoarseGrainedExecutorBackend(
hostname: String,
cores: Int,
userClassPath: Seq[URL],
- env: SparkEnv)
+ env: SparkEnv,
+ resourcesFile: Option[String])
Review comment:
Yes, the intention is for this to be used by standalone mode. It has no
support for containers or isolation like the other resource managers have so it
will pass in a file specifying which resources for that container to use. In
the GPU case if the node has 4 gpu's and the executor asked for one, it would
pass in one GPU resource and the id of that 1 gpu it should use. There is a
separate jira to add that to standalone mode.
The other resources managers I believe all support some sort of container
and isolation such that when the executor starts up it can just use the
discoveryScript to auto discover what resources it has. We have tested on
kubernetes and yarn, I don't have a mesos setup to test on but someone I
believe volunteered to work on that side.
You can obviously run other resources manager like YARN without docker and
isolation and in that case this wouldn't work, but YARN doesn't give you any
other option. It doesn't tell you what GPU you received for instance, so in
the documentation we will put the limitations and required that
yarn/kubernetes, etc must be configured in such a way for this to work. If you
are running YARN without isolation then you could also come up with discovery
script to make this work, similar to the hacks users are doing now on YARN with
gpu scheduling.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
