tgravescs commented on a change in pull request #27313: [SPARK-29148][CORE] Add
stage level scheduling dynamic allocation and scheduler backend changes
URL: https://github.com/apache/spark/pull/27313#discussion_r375985031
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -69,27 +69,17 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
conf.get(SCHEDULER_MAX_REGISTERED_RESOURCE_WAITING_TIME))
private val createTimeNs = System.nanoTime()
- private val taskResourceNumParts: Map[String, Int] =
- if (scheduler.resourcesReqsPerTask != null) {
- scheduler.resourcesReqsPerTask.map(req => req.resourceName ->
req.numParts).toMap
- } else {
- Map.empty
- }
-
// Accessing `executorDataMap` in the inherited methods from
ThreadSafeRpcEndpoint doesn't need
// any protection. But accessing `executorDataMap` out of the inherited
methods must be
// protected by `CoarseGrainedSchedulerBackend.this`. Besides,
`executorDataMap` should only
// be modified in the inherited methods from ThreadSafeRpcEndpoint with
protection by
// `CoarseGrainedSchedulerBackend.this`.
private val executorDataMap = new HashMap[String, ExecutorData]
- // Number of executors requested by the cluster manager,
[[ExecutorAllocationManager]]
- @GuardedBy("CoarseGrainedSchedulerBackend.this")
- private var requestedTotalExecutors = 0
-
- // Number of executors requested from the cluster manager that have not
registered yet
+ // Number of executors for each ResourceProfile requested by the cluster
+ // manager, [[ExecutorAllocationManager]]
@GuardedBy("CoarseGrainedSchedulerBackend.this")
- private var numPendingExecutors = 0
Review comment:
when you say it was used, are you just talking about logging, or am I
missing something? All I see it being used for is debug logging, it isn't
actually preventing any code paths. Keeping it adds a lot of overhead with the
additional profiles so I chose to remove it vs getting the extra debug
information. I can investigate more at adding it back, I just remember I was
going to have to modify more then I wanted to keep it. But if its really
useful I can, I just don't remember using it in production anytime recently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
