andygrove opened a new pull request, #2655:
URL: https://github.com/apache/datafusion-ballista/pull/2655
# Which issue does this PR close?
Closes #2650.
# Rationale for this change
Under `--scheduler-policy pull-staged` the scheduler sizes each task to the
free vcores the executor reports, and `bind_one` charges the task one vcore per
partition packed into it. The executor took one permit per task however many
partitions it carried, so after an 8-partition task an 8-vcore executor still
reported 7 vcores free and its next poll took another task. Executors that
polled later could be left with no work, and because each task's memory pool is
sized from `vcores_consumed`, the memory budget was overcommitted the same way.
#2650 has the full analysis and the SF1000 numbers.
Push-staged is not affected: the scheduler tracks vcores itself there and
refunds `vcores_consumed` when each task completes.
# What changes are included in this PR?
- `poll_loop` acquires `vcores_consumed` permits per task
(`acquire_many_owned`) and holds them until the task finishes, as the single
permit was held before. The free count the next poll reports is then the number
of vcores not in use.
- The count is clamped to between 1 and the semaphore's size, read at
startup. At least 1 because a scheduler that predates the field sends 0. At
most the size because `acquire_many_owned` on more permits than the semaphore
will ever hold never returns, which would stall the loop and with it the
heartbeat. Hitting the upper clamp logs a warning. A well-behaved scheduler
never hits it, since it sizes tasks to the free count the executor reported.
- Three tests drive `poll_loop` against an in-process mock scheduler
(`SchedulerGrpcServer` called directly, so no sockets) and a query stage that
blocks until released, and read the semaphore while the task runs. Each also
checks that the permits come back when the task finishes.
- An 8-partition task on 8 vcores leaves 0 free (7 before this change). A
3-partition task leaves 5.
- A task charged 1 vcore, or 0 as an older scheduler sends, holds one
permit.
- A task charged 100 vcores on an 8-vcore executor starts and holds all 8,
rather than hanging.
The first and last fail without the change. The middle one guards the
lower clamp.
# Are there any user-facing changes?
No API changes: `poll_loop`'s signature is unchanged.
Pull-staged executors, including the standalone one, no longer run more
partitions than they have vcores, so work spreads across the executors that
poll. When a memory pool size is set, the per-task pools no longer add up to
more than the executor's budget.
A task keeps all of its vcores until its slowest partition finishes, which
is how push-staged already works, so a straggler partition leaves the others'
vcores idle until it ends. Releasing permits per partition would be a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
