Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7243#discussion_r34062845
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -218,11 +218,12 @@ private[spark] class ApplicationMaster(
}
}
- private def sparkContextInitialized(sc: SparkContext) = {
+ private def sparkContextInitialized(sc: SparkContext, backend:
CoarseGrainedSchedulerBackend) = {
sparkContextRef.synchronized {
sparkContextRef.compareAndSet(null, sc)
sparkContextRef.notifyAll()
}
+ allocator.setScheduler(backend)
--- End diff --
How does this work? This method is called before the `allocator` field is
initialized. From `runDriver`, for example:
// This a bit hacky, but we need to wait until the spark.driver.port
property has
// been set by the Thread executing the user class.
val sc = waitForSparkContextInitialized()
// If there is no SparkContext at this point, just fail the app.
if (sc == null) {
...
} else {
....
registerAM(rpcEnv, sc.ui.map(_.appUIAddress).getOrElse(""),
securityMgr) /* this is where allocator is initialized */
...
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
