dongjoon-hyun commented on a change in pull request #27697: [SPARK-27750]
Standalone scheduler - ability to prioritize applications over drivers, many
drivers act like Denial of Service
URL: https://github.com/apache/spark/pull/27697#discussion_r385962328
##########
File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
##########
@@ -797,28 +798,32 @@ private[deploy] class Master(
val shuffledAliveWorkers = Random.shuffle(workers.toSeq.filter(_.state ==
WorkerState.ALIVE))
val numWorkersAlive = shuffledAliveWorkers.size
var curPos = 0
- for (driver <- waitingDrivers.toList) { // iterate over a copy of
waitingDrivers
- // We assign workers to each waiting driver in a round-robin fashion.
For each driver, we
- // start from the last worker that was assigned a driver, and continue
onwards until we have
- // explored all alive workers.
- var launched = false
- var isClusterIdle = true
- var numWorkersVisited = 0
- while (numWorkersVisited < numWorkersAlive && !launched) {
- val worker = shuffledAliveWorkers(curPos)
- isClusterIdle = worker.drivers.isEmpty && worker.executors.isEmpty
- numWorkersVisited += 1
- if (canLaunchDriver(worker, driver.desc)) {
- val allocated = worker.acquireResources(driver.desc.resourceReqs)
- driver.withResources(allocated)
- launchDriver(worker, driver)
- waitingDrivers -= driver
- launched = true
+ val allFreeCores = shuffledAliveWorkers.map(_.coresFree).sum
+ val forDriversFreeCores = math.max(allFreeCores - coresReservedForApps, 0)
+ if (forDriversFreeCores > 0) {
Review comment:
Could you make a test case for this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
