andygrove opened a new issue, #2716: URL: https://github.com/apache/datafusion-ballista/issues/2716
## Is your feature request related to a problem or challenge? Pipelined execution (#2652, #2657) places every task of a job at once, so a pipelined job waits until the vcores of its widest stage are free on the cluster all at the same moment. Admission today has two gaps: - **Starvation behind blocking jobs.** `revive_offers` tries pipelined jobs first, but a job that doesn't fit simply waits, and the same pass then binds the free vcores to blocking jobs one task at a time. If blocking work keeps arriving, enough vcores may never be free at once and the pipelined job waits indefinitely. - **No ordering between pipelined jobs.** Pipelined jobs are tried in the iteration order of `TaskManager::get_running_job_cache`, which is a `HashMap`. A wide job can keep waiting behind narrower jobs that keep fitting. #2652 lists fair admission between pipelined and blocking jobs as a follow-up and answers its open question 3 with "wait". ## Describe the solution you'd like Some form of fair admission, for example: - try waiting pipelined jobs in submission order (FIFO by `queued_at`), and - once the oldest waiting pipelined job has waited past some threshold, stop binding new blocking tasks onto the vcores it needs, so they drain and it can be placed. ## Describe alternatives you've considered Falling back to blocking execution when a pipelined job can't be placed within some time (the other option from #2652 question 3). ## Additional context Raised in review of #2657. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
