tgravescs commented on a change in pull request #26682: [SPARK-29306][CORE]
Stage Level Sched: Executors need to track what ResourceProfile they are
created with
URL: https://github.com/apache/spark/pull/26682#discussion_r371362785
##########
File path: core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
##########
@@ -124,6 +124,35 @@ private[spark] object ResourceUtils extends Logging {
.filter(_.amount > 0)
}
+ // Used to take a fraction amount from a task resource requirement and split
into a real
+ // integer amount and the number of parts expected. For instance, if the
amount is 0.5,
+ // the we get (1, 2) back out.
+ // Returns tuple of (amount, numParts)
+ def calculateAmountAndPartsForFraction(amount: Double): (Int, Int) = {
+ val parts = if (amount <= 0.5) {
+ Math.floor(1.0 / amount).toInt
+ } else if (amount % 1 != 0) {
+ throw new SparkException(
+ s"The resource amount ${amount} must be either <= 0.5, or a whole
number.")
+ } else {
+ 1
Review comment:
note it ends up being used stored here:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala#L64
It ultimately ends up being used here:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/resource/ResourceAllocator.scala#L33
where we track the allocations of each resource. this allows for a single
address to be used multiple times
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
