vanzin commented on a change in pull request #19616: [SPARK-22404][YARN]
Provide an option to use unmanaged AM in yarn-client mode
URL: https://github.com/apache/spark/pull/19616#discussion_r243433749
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -324,6 +308,59 @@ private[spark] class ApplicationMaster(args:
ApplicationMasterArguments) extends
}
}
+ def runUnmanaged(clientRpcEnv: RpcEnv,
+ appAttemptId: ApplicationAttemptId,
+ stagingDir: Path): Unit = {
+ try {
+ new CallerContext(
+ "APPMASTER", sparkConf.get(APP_CALLER_CONTEXT),
+ Option(appAttemptId.getApplicationId.toString),
None).setCurrentContext()
+
+ // This shutdown hook should run *after* the SparkContext is shut down.
Review comment:
This is client mode, so you can't rely on shutdown hooks. You need to
explicitly stop this service when the SparkContext is shutdown.
Imagine someone just embeds `sc = new SparkContext(); ...; sc.stop()` in
their app code, but the app itself runs for way longer than the Spark app.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
