Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19616#discussion_r165518653
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
---
@@ -1104,14 +1117,39 @@ private[spark] class Client(
if (returnOnRunning && state == YarnApplicationState.RUNNING) {
return (state, report.getFinalApplicationStatus)
}
-
+ if (state == YarnApplicationState.ACCEPTED &&
isClientUnmanagedAMEnabled
+ && !amServiceStarted && report.getAMRMToken != null) {
+ amServiceStarted = true
+ startApplicationMasterService(report)
+ }
lastState = state
}
// Never reached, but keeps compiler happy
throw new SparkException("While loop is depleted! This should never
happen...")
}
+ private def startApplicationMasterService(report: ApplicationReport) = {
+ // Add AMRMToken to establish connection between RM and AM
+ val token = report.getAMRMToken
+ val amRMToken:
org.apache.hadoop.security.token.Token[AMRMTokenIdentifier] =
+ new org.apache.hadoop.security.token.Token[AMRMTokenIdentifier](token
+ .getIdentifier().array(), token.getPassword().array, new Text(
+ token.getKind()), new Text(token.getService()))
+ val currentUGI = UserGroupInformation.getCurrentUser
+ currentUGI.addToken(amRMToken)
+
+ System.setProperty(
+ ApplicationConstants.Environment.CONTAINER_ID.name(),
+ ContainerId.newContainerId(report.getCurrentApplicationAttemptId,
1).toString)
+ val amArgs = new ApplicationMasterArguments(Array("--arg",
--- End diff --
This is pretty weird, I'd make this an explicit constructor argument for
the AM instead. But if I understand this correctly, this is the address the AM
will be connecting back to the driver, right?
It seems like there's an opportunity for better code here, since now they'd
both be running in the same process. Like in the cluster mode case, where the
AM uses the same `RpcEnv` instance as the driver (see `runDriver()`).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
