Dandandan opened a new pull request, #2904: URL: https://github.com/apache/datafusion-ballista/pull/2904
# Which issue does this PR close? No issue filed yet. # Rationale for this change A task that packs N partitions is charged N vcores and holds all of them until its slowest partition ends. When partitions are skewed (one large file or one hot key next to many small ones), most of those vcores sit idle for most of the task, while other stages and jobs wait for an executor with free vcores. This PR hands a partition's vcore back as soon as that partition finishes, so the scheduler can bind new work to it while the rest of the task keeps running. # What changes are included in this PR? - **`PartitionVcoreReleaser`** (`ballista/core/src/execution_plans/vcore_release.rs`): attached to a task's session config by the executor. A task needs `min(running partitions, vcores)` vcores and keeps at least one until it ends; the rest is released as partitions complete. - **Shuffle writers** (`ShuffleWriterExec`, `SortShuffleWriterExec`, `RangeShuffleWriterExec`) report each input partition they finish draining. - **Pull mode**: the task splits the finished partitions' permits off its semaphore permit, so the next `poll_work` reports them as free. No protocol change is needed. - **Push mode**: the executor sends a `RunningTask` status carrying the new `vcores_released` field. The scheduler refunds those vcores from the task's entry in `outstanding_task_reservations` and leaves at least one for the terminal status, which takes what is left. Because every refund goes through that one ledger, a lost, late or repeated running report cannot refund a vcore twice, and abort and launch-failure paths refund only what the task still holds. Running reports are dropped before they reach the execution graph. - **`ballista.scheduler.release_partition_vcores`** (default `true`) turns it off per session. Collapse stages hold one vcore and pipelined jobs reserve vcores per job, so neither releases anything. Things worth a reviewer's attention: - **Memory**: a task's memory pool is sized by the vcores it was charged and is not shrunk when vcores are released. While a task with released vcores is still running, the task pools on an executor can add up to more than its memory. Finished partitions have stopped allocating, but the remaining ones can still use the whole pool. - **Status traffic in push mode**: one running status per released vcore (batched by the existing status reporter). - Executors only send running reports to a scheduler of the same `BALLISTA_PROTOCOL_VERSION`, which is already at 4 on `main` and unreleased, so I did not bump it again. # Are there any user-facing changes? A new session setting, `ballista.scheduler.release_partition_vcores`, on by default. `RunningTask` gains a `vcores_released` field (`proto-change`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
