Dandandan opened a new issue, #2905:
URL: https://github.com/apache/datafusion-ballista/issues/2905

   **Is your feature request related to a problem or challenge? Please describe 
what you are trying to do.**
   
   A task that packs N partitions is charged N vcores (`bind_one`) and holds 
all of them until its slowest partition ends. When the partitions of a task are 
skewed, for example one large file or one hot key next to many small ones, most 
of those vcores sit idle for most of the task. Meanwhile other stages and other 
jobs wait for an executor with free vcores, so the cluster looks full while a 
good part of it is doing nothing.
   
   This got more visible when `ballista.scheduler.max_partitions_per_task` 
started defaulting to `0` (unbounded): a single task now commonly holds every 
free vcore of an executor.
   
   **Describe the solution you'd like**
   
   Hand a partition's vcore back as soon as that partition finishes, while the 
rest of the task keeps running:
   
   - The shuffle writers report each input partition they finish draining.
   - A task needs `min(running partitions, vcores)` vcores and keeps at least 
one until it ends.
   - Pull mode: the task releases the finished partitions' semaphore permits, 
so the next `poll_work` reports them as free.
   - Push mode: the executor reports the released vcores in a running task 
status, and the scheduler refunds them from the task's reservation. The 
terminal status returns whatever is left.
   - A session setting turns it off.
   
   Collapse stages hold a single vcore and pipelined jobs reserve vcores per 
job, so neither is affected.
   
   **Describe alternatives you've considered**
   
   - Lowering `ballista.scheduler.max_partitions_per_task`. Smaller tasks free 
their vcores sooner, but every stage then pays for more tasks whether its 
partitions are skewed or not, and a task of several partitions still waits for 
its slowest one.
   - Splitting skewed partitions at planning time. This helps where the skew is 
known up front, but not when it only shows at run time.
   
   **Additional context**
   
   Open questions:
   
   - A task's memory pool is sized by the vcores it was charged and is not 
shrunk when vcores are released, so the task pools on an executor can 
temporarily add up to more than its memory.
   - Vcores freed mid-stage are bound to that stage's remaining partitions as 
extra small tasks, which may cost more than it gains for stages without skew.
   - Pipelined jobs could release from their per-job reservation in a similar 
way, as a follow-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to