andygrove opened a new issue, #2650: URL: https://github.com/apache/datafusion-ballista/issues/2650
> Disclaimer: this issue was drafted by an LLM (Claude Code) at my request, against `b89a4e46`. File/line references were checked against the code, but please treat the analysis as a suggestion and push back where it's wrong. **Describe the bug** Under `--scheduler-policy pull-staged`, an executor can run several times more partitions than it has vcores, while executors that poll later get no work. The cause is a mismatch in vcore accounting. The scheduler sizes each task to the free vcores the executor reports. The executor then charges each task one permit, however many partitions the task carries: 1. `poll_loop` reports `num_free_vcores: free_vcores.available_permits()` ([execution_loop.rs#L173](https://github.com/apache/datafusion-ballista/blob/b89a4e4654b0bd5259856600d0e45703326b9d3f/ballista/executor/src/execution_loop.rs#L173)). 2. `poll_work` turns that number into a single budget and calls `bind_task_bias`, `bind_task_round_robin` or the custom policy ([grpc.rs#L177-L195](https://github.com/apache/datafusion-ballista/blob/b89a4e4654b0bd5259856600d0e45703326b9d3f/ballista/scheduler/src/scheduler_server/grpc.rs#L177-L195)). 3. For a non-collapse stage, `bind_one` packs up to `min(budget.vcores, max_partitions_per_task)` partitions into one task and records `vcores_consumed = slice.len()` ([cluster/mod.rs#L537-L622](https://github.com/apache/datafusion-ballista/blob/b89a4e4654b0bd5259856600d0e45703326b9d3f/ballista/scheduler/src/cluster/mod.rs#L537-L622)). 4. The executor acquires one permit for that task, not `vcores_consumed` permits ([execution_loop.rs#L207-L211](https://github.com/apache/datafusion-ballista/blob/b89a4e4654b0bd5259856600d0e45703326b9d3f/ballista/executor/src/execution_loop.rs#L207-L211)). On an executor with 8 vcores, this plays out as follows: - The first poll returns an 8-partition task and leaves 7 permits. - The next poll reports 7 free vcores and returns a 7-partition task, and so on. - The executor can end up running 8 + 7 + … + 1 = 36 partitions on 8 vcores. Memory is overcommitted the same way. Each task's memory pool is sized from `vcores_consumed`: `run_received_task` passes it to `produce_runtime_for_session` ([execution_loop.rs#L348-L352](https://github.com/apache/datafusion-ballista/blob/b89a4e4654b0bd5259856600d0e45703326b9d3f/ballista/executor/src/execution_loop.rs#L348-L352)), which multiplies it by a per-vcore share ([executor_process.rs#L250](https://github.com/apache/datafusion-ballista/blob/b89a4e4654b0bd5259856600d0e45703326b9d3f/ballista/executor/src/executor_process.rs#L250)). Those 36 partitions are therefore promised 4.5× the executor's memory budget. One permit per task was correct when a task was one partition. These lines weren't updated when tasks became multi-partition in #2038. Push-staged scheduling is not affected, because there the scheduler tracks vcores itself and refunds `vcores_consumed` when each task completes. **Observed** This came up in a TPC-H SF1000 run on 32 executors with 8 vcores each, under pull-staged: - In Q1's scan stage, one executor received tasks of 8, 7, 6, 5, 4 and 3 partitions (33 partitions on 8 vcores), and 15 of the 32 executors received no tasks. - Across the run, 111 of 150 large stages had an executor running 13 to 36 partitions. "Large" here means at least 200 partitions and a median task time over 1 s. **Expected behavior** An executor never runs more partitions concurrently than it has vcores, so work spreads across the executors that poll. **Suggested fix** - In `poll_loop`, acquire `task.vcores_consumed.max(1)` permits per task (for example `acquire_many_owned`) and hold them for the task's lifetime, as the single permit is held today. - Add a test that drives `poll_loop` against a mock scheduler returning an 8-partition task, and asserts that the next poll reports 0 free vcores. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
