ivoson commented on code in PR #36716:
URL: https://github.com/apache/spark/pull/36716#discussion_r902839471
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala:
##########
@@ -402,7 +420,7 @@ object ResourceProfile extends Logging {
}
private[spark] case class ExecutorResourcesOrDefaults(
- cores: Int,
+ cores: Option[Int], // Can only be None for standalone cluster.
Review Comment:
Thanks, done.
##########
docs/spark-standalone.md:
##########
@@ -455,6 +455,14 @@ if the worker has enough cores and memory. Otherwise, each
executor grabs all th
on the worker by default, in which case only one executor per application may
be launched on each
worker during one single schedule iteration.
+# Stage Level Scheduling Overview
+
+Stage level scheduling is supported on Standalone when dynamic allocation is
enabled. Currently, when Master allocates executors for one application, will
schedule based on the order of the ResourceProfile ids for multiple
ResourceProfiles. The ResourceProfile with smaller id will be scheduled
firstly. Normally this won’t matter as Spark finishes one stage before starting
another one, the only case this might have an affect is in a job server type
scenario, so its something to keep in mind. For scheduling, we will only take
executor memory and executor cores from built-in executor resources and all
other custom resources from a ResourceProfile, other built-in executor
resources such as offHeap and memoryOverhead won't take any effect. The base
default profile will be created based on the spark configs when you submit an
application. Executor memory and executor cores from the base default profile
can be propagated to custom ResourceProfiles, but all other custom resources
can not
be propagated.
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
