tgravescs commented on a change in pull request #26682: [SPARK-29306][CORE]
Stage Level Sched: Executors need to track what ResourceProfile they are
created with
URL: https://github.com/apache/spark/pull/26682#discussion_r371324318
##########
File path: core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
##########
@@ -124,6 +124,35 @@ private[spark] object ResourceUtils extends Logging {
.filter(_.amount > 0)
}
+ // Used to take a fraction amount from a task resource requirement and split
into a real
+ // integer amount and the number of parts expected. For instance, if the
amount is 0.5,
+ // the we get (1, 2) back out.
+ // Returns tuple of (amount, numParts)
+ def calculateAmountAndPartsForFraction(amount: Double): (Int, Int) = {
+ val parts = if (amount <= 0.5) {
+ Math.floor(1.0 / amount).toInt
+ } else if (amount % 1 != 0) {
+ throw new SparkException(
+ s"The resource amount ${amount} must be either <= 0.5, or a whole
number.")
+ } else {
+ 1
Review comment:
Sure it needs more documentation then, I'll update the comment to have more
context and perhaps clarify the names a bit.
This is used for task resources (doesn't apply to executor resources, only
task level and only supported when value < 1, we don't allow 2.33 for instance)
when you have fractional amounts. For instance you can set
spark.task.resource.gpu.amount=0.33 and what this really means is that I want 3
tasks to use the same GPU.
so do that inside Spark we convert this into an amount and a number of parts
per resource. Really number of parts you can think of as the number of slots
per GPU address. For 0.33 the amount = 1 and the numparts becomes 3. If the
spark.task.resource.gpu.amount=2 the amount =2 and numparts = 1. So numparts
is the number of ways to subdivide a resource address when the amount = 1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
