tgravescs commented on a change in pull request #27313: [SPARK-29148][CORE] Add
stage level scheduling dynamic allocation and scheduler backend changes
URL: https://github.com/apache/spark/pull/27313#discussion_r377313125
##########
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##########
@@ -726,14 +894,23 @@ private[spark] class ExecutorAllocationManager(
})
}
- registerGauge("numberExecutorsToAdd", numExecutorsToAdd, 0)
+ // The metrics are going to return the numbers for the default
ResourceProfile.
+ // It would be nice to do include each profile somehow in the future.
Review comment:
No the definition is not documented. Not sure why it never was.
I personally think this metric is odd because you have to understand how the
algorithm works for this number to make sense.
Its not total outstanding requests either, its the number to add on the next
iteration (not the one you are asking for now) if you still need executors.
And just because its the number to add the next iteration doesn't mean you are
going to really ask for that many.
The numberTargetExecutors is what you are currently asking for.
I agree with you that doing totals here make sense and then in the future
having per profile ones that add up to the total. In this case I think that
number would be useless though.
Really I'm almost more in favor of removing it as I don't really see it
being very useful as a metric. thoughts on that?
I'll update the PR to do a sum for all of them and then can change/remove
based on your thoughts
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
