tgravescs commented on a change in pull request #26682: [SPARK-29306][CORE]
Stage Level Sched: Executors need to track what ResourceProfile they are
created with
URL: https://github.com/apache/spark/pull/26682#discussion_r371519391
##########
File path: core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
##########
@@ -124,6 +124,35 @@ private[spark] object ResourceUtils extends Logging {
.filter(_.amount > 0)
}
+ // Used to take a fraction amount from a task resource requirement and split
into a real
+ // integer amount and the number of parts expected. For instance, if the
amount is 0.5,
+ // the we get (1, 2) back out.
+ // Returns tuple of (amount, numParts)
+ def calculateAmountAndPartsForFraction(amount: Double): (Int, Int) = {
+ val parts = if (amount <= 0.5) {
+ Math.floor(1.0 / amount).toInt
+ } else if (amount % 1 != 0) {
+ throw new SparkException(
+ s"The resource amount ${amount} must be either <= 0.5, or a whole
number.")
+ } else {
+ 1
Review comment:
> If gpus on an executor is 2 and spark.task.resource.gpu.amount=0.33, is it
invalid configuration
Yes its valid, executor has 2 gpus and you allow 3 tasks on each gpu, means
you can run 6 total tasks in parallel (assuming you have 6 cores and
spark.task.cpus=1).
> If spark.task.resource.gpu.amount=1 and executor has 2 gpu's, does it mean
the task needs both gpu's ? Or only 1 ?
It means each task gets 1 gpu, so if executor has 2 gpus you can run 2
tasks.
if you have spark.task.resource.gpu.amount > 1 that means each task gets
that amount of GPUs. Meaning executor gpus=4 and
spark.task.resource.gpu.amount=2, means each task gets 2 gpus and you have 2
tasks total on that executor.
If you configure spark.task.resource.gpu.amount=4 and
spark.executor.resource.gpu.amount=1 - spark throws an error as its invalid
configuration.
The idea was spark.task.resource.gpu.amount acts just like spark.task.cpus -
with the addition it also supports the fractional amounts in case you want to
run multiple tasks on a single gpu.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
