tgravescs commented on a change in pull request #24374: [SPARK-27366][CORE]
Support GPU Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#discussion_r287381653
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -335,9 +339,10 @@ private[spark] class TaskSchedulerImpl(
for (i <- 0 until shuffledOffers.size) {
val execId = shuffledOffers(i).executorId
val host = shuffledOffers(i).host
- if (availableCpus(i) >= CPUS_PER_TASK) {
+ if (availableCpus(i) >= CPUS_PER_TASK &&
+ resourceMeetTaskRequirements(availableResources(i))) {
try {
- for (task <- taskSet.resourceOffer(execId, host, maxLocality)) {
+ for (task <- taskSet.resourceOffer(execId, host, maxLocality,
availableResources(i))) {
tasks(i) += task
val tid = task.taskId
taskIdToTaskSetManager.put(tid, taskSet)
Review comment:
I'd be ok with a comment for now. I wasn't suggesting only book keeping by
count, I meant do the actual acquire addresses here, but you would have to make
other changes for that to happen.
One thing we could do is make a separate function in TaskSchedulerImpl for
the resource (including cpu's) accounting and then have
TaskSetManager.resourceOffer call into that to acquire the resources.
Another option is to have the TaskSetManager get some addresses but only do
the actual accounting to remove them from the available here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
