GitHub user scwf opened a pull request:

    [SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSchedulerBackend reset

    ## What changes were proposed in this pull request?
    Now `CoarseGrainedSchedulerBackend` reset will get the lock,
      protected def reset(): Unit = synchronized {
        numPendingExecutors = 0
        // Remove all the lingering executors that should be removed but not 
yet. The reason might be
        // because (1) disconnected event is not yet received; (2) executors 
die silently.
        executorDataMap.toMap.foreach { case (eid, _) =>
            RemoveExecutor(eid, SlaveLost("Stale executor after cluster manager 
     but on removeExecutor also need the lock 
"CoarseGrainedSchedulerBackend.this.synchronized", this will cause deadlock.
       private def removeExecutor(executorId: String, reason: 
ExecutorLossReason): Unit = {
          logDebug(s"Asked to remove executor $executorId with reason $reason")
          executorDataMap.get(executorId) match {
            case Some(executorInfo) =>
              // This must be synchronized because variables mutated
              // in this block are read when requesting executors
              val killed = CoarseGrainedSchedulerBackend.this.synchronized {
                addressToExecutorId -= executorInfo.executorAddress
                executorDataMap -= executorId
                executorsPendingLossReason -= executorId
    ## How was this patch tested?
    manual test.

You can merge this pull request into a Git repository by running:

    $ git pull spark-17929

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15481
commit 3681fae6b5364a5cf55700e1510473d8d9b77cd3
Author: w00228970 <>
Date:   2016-10-14T09:24:30Z

    use send


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to