liupc opened a new pull request #27862: [SPARK-31107][CORE]Extend FairScheduler to support pool level resource isolation URL: https://github.com/apache/spark/pull/27862 ### What changes were proposed in this pull request? Currently, spark only provided two types of scheduler: FIFO & FAIR, but in sql high-concurrency scenarios, a few of drawbacks are exposed. FIFO: it can easily causing congestion when large sql query occupies all the resources FAIR: the taskSets of one pool may occupies all the resource due to there are no hard limit on the maximum usage for each pool. this case may be frequently met under high workloads. So we propose to add a maxShare argument for FairScheduler to control the maximum running tasks for each pool. ### Why are the changes needed? Improvements ### Does this PR introduce any user-facing change? No ### How was this patch tested? existing UT
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
