Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5144#discussion_r28018296
--- Diff: core/src/main/scala/org/apache/spark/deploy/rest/RestClient.scala
---
@@ -219,14 +221,20 @@ private[deploy] class StandaloneRestClient extends
Logging {
/** Return the base URL for communicating with the server, including the
protocol version. */
private def getBaseUrl(master: String): String = {
- val masterUrl = master.stripPrefix("spark://").stripSuffix("/")
+ val masterUrl = if (master.startsWith("spark://")) {
+ master.stripPrefix("spark://").stripSuffix("/")
+ } else {
+ master.stripPrefix("mesos://").stripSuffix("/")
+ }
--- End diff --
This logic is duplicated here and in Spark submit. Maybe it makes sense to
have
```
private val supportedMasterPrefixes = Seq("spark://", "mesos://")
private def getBaseUrl(master: String): String = {
var masterUrl = master
supportedMasterPrefixes.foreach { prefix =>
if (master.startsWith(prefix)) {
masterUrl = master.stripPrefix(prefix)
}
}
masterUrl = masterUrl.stripSuffix("/")
s"http://$masterUrl/...";
}
def validateMaster(master: String): Unit = {
val valid = supportedMasterPrefixes.exists { prefix =>
master.startsWith(prefix) }
if (!valid) {
throw new IllegalArgumentException(
"This REST client only supports master URLs that start with " +
"one of the following: " + supportedMasterPrefixes.mkString(","))
}
}
```
Then Spark submit can just call `validateMaster`. For that you might need
to make this method static.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
