Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20807#discussion_r175151438
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -418,7 +418,19 @@ private[spark] class ApplicationMaster(args:
ApplicationMasterArguments) extends
}
private def sparkContextInitialized(sc: SparkContext) = {
+ // Notify runDriver function that SparkContext is available
sparkContextPromise.success(sc)
+ // Pause the user class thread in order to make proper initialization
in runDriver function.
+ // When it happened the thread has to be resumed with resumeDriver
function.
+ sparkContextPromise.synchronized {
--- End diff --
Hmm, there's a race now.
You're updating the promise outside the lock, so it's possible that the
`runDriver` thread can see that and notify the lock before this thread grabs
it, so this thread would hang forever in that case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
