ivoson commented on code in PR #36716:
URL: https://github.com/apache/spark/pull/36716#discussion_r885749761
##########
core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala:
##########
@@ -65,7 +66,70 @@ private[spark] class ApplicationInfo(
appSource = new ApplicationSource(this)
nextExecutorId = 0
removedExecutors = new ArrayBuffer[ExecutorDesc]
- executorLimit = desc.initialExecutorLimit.getOrElse(Integer.MAX_VALUE)
+ val initialExecutorLimit =
desc.initialExecutorLimit.getOrElse(Integer.MAX_VALUE)
+
+ rpIdToResourceProfile = new mutable.HashMap[Int, ResourceProfile]()
+ rpIdToResourceProfile(DEFAULT_RESOURCE_PROFILE_ID) = desc.defaultProfile
+ rpIdToResourceDesc = new mutable.HashMap[Int, ResourceDescription]()
+ createResourceDescForResourceProfile(desc.defaultProfile)
+
+ targetNumExecutorsPerResourceProfileId = new mutable.HashMap[Int, Int]()
+ targetNumExecutorsPerResourceProfileId(DEFAULT_RESOURCE_PROFILE_ID) =
initialExecutorLimit
Review Comment:
Not sure if I get this right. Please also help check this. In
[ApplicationInfo](https://github.com/apache/spark/blob/v3.3.0-rc3/core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala#L49)
we can see that the variable `initialExecutorLimit` is used for dynamic
allocation to limit the executor num for the app to start with.
And when [dynamic allocation is
enabled](https://github.com/apache/spark/blob/v3.3.0-rc3/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala#L120),
the limit would be 0, otherwise it would be None for now. And the
`ApplicationDescription` is created during `SparkContext` initialization. At
this time point, we can only have default resource profile.
For other resource profiles, we rely on
[`ExecutorAllocationManager`](https://github.com/apache/spark/blob/v3.3.0-rc3/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L431)
to update the requests for different resource profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
