[
https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212820#comment-16212820
]
ASF GitHub Bot commented on GEARPUMP-355:
-----------------------------------------
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146002316
--- Diff:
experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala
---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system:
ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system:
ActorSystem): ActorRef = {
+ val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+ LOG.info(s"appMasterPath=$appMasterPath")
+
+ val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+ val url: URL = new URL(appMasterPath)
+ val connection: HttpURLConnection =
connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+ connection.setInstanceFollowRedirects(true)
+
+ try {
+ connection.connect()
+ } catch {
+ case e: IOException =>
+ LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+ }
+
+ val status = connection.getResponseCode
+ if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+ } else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please
make sure " +
+ s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
should this be `getTrackingUrl` now ?
> AppMasterResolver fails to run against a kerberized Hadoop cluster
> ------------------------------------------------------------------
>
> Key: GEARPUMP-355
> URL: https://issues.apache.org/jira/browse/GEARPUMP-355
> Project: Apache Gearpump
> Issue Type: Bug
> Components: security, yarn
> Affects Versions: 0.8.4
> Reporter: Timea Magyar
> Assignee: Timea Magyar
> Fix For: 0.8.4
>
>
> When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn
> environment, after the Application Master address has been resolved as a
> prerequisite, the YarnAppMaster (responsible for starting GearPump masters,
> workers, UI servers as Yarn containers) address (actor reference) must be
> obtained via Kerberos/Spnego. (Kerberos over http)
> The current implementation for this resides in the AppMasterResolver class
> and is using an apache http client (version 3.x) for establishing a
> connection to the Application Master and obtain the above YarnAppMaster actor
> reference. Since the apache http client does not support the negotiate
> authentication scheme in version 3.x (required for a connection over
> kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop
> cluster set-up.
> I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to
> provide a patch for this by adapting the SPNEGO-enabled Hadoop web
> connection code from WebHDFS.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
