[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf closed the pull request at: https://github.com/apache/spark/pull/1980 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-95577333 i am closing this in favor of #5664 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-72575398 Hi @scwf, I just merged #3571, which added SSL support for Akka and the HTTPServer. That PR added a common configuration mechanism for SSL, plus some useful utilities for testing this functionality. If you're interested, it would be great to revisit HTTPS for the web UI now that we've merged that other patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-72580281 @scwf Yep, that PR only added SSL support to Akka and the HTTP FileServer, not the web UI or the shuffle service. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-72580055 in that PR now no https support for web ui right? if so i will rework this to support it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-60173980 @scwf can you take a look at https://github.com/apache/spark/issues/2739? I created PR which aim is to add SSL support for Akka based connections and for HttpServer which in turn bases on Jetty. I use SSLOptions object to configure SSL. All the settings are being kept in a separate file `ssl.conf`. You can use different name spaces to separate configurations for say UI, control and data. Would you take a look? It would be nice to have a single place to configure SSL. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-60174311 Btw. I was trying to do SSL for UI as well. One thing I'm afraid of is that this solution would break any recovery information for Spark Master (different signature of WorkerInfo). Also, it doesn't allow to run the cluster in mixed mode during upgrade - some nodes running older Spark and some nodes running newer Spark, because it changes the signature of messages. Anyway, this is very minor I think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-60180284 @jacek-lewandowski, you mean in ```ssl.conf``` to set ssl configs for ui( and for akka, httpserver)? then ssl.con f is the single place to configure ssl, right? if so, i think it's ok. For the second one some nodes running older Spark and some nodes running newer Spark, i don't think there is this situation. @JoshRosen @vanzin is this ok to go? I think we can merge this first then in #2739 to unify the ssl configs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18692553
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ // to generate a new url string scheme://server:port+path
--- End diff --
Ok, updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58624892 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21584/consoleFull) for PR 1980 at commit [`3b01d3a`](https://github.com/apache/spark/commit/3b01d3a66d21854b5e053de89f3d301fee58b315). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58636687 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21584/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58636685 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21584/consoleFull)** for PR 1980 at commit [`3b01d3a`](https://github.com/apache/spark/commit/3b01d3a66d21854b5e053de89f3d301fee58b315) after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58672706 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21595/consoleFull) for PR 1980 at commit [`4ae834b`](https://github.com/apache/spark/commit/4ae834b6c5aafc36d44adba4c4dbb934a63ea3ed). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58681640 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21595/consoleFull) for PR 1980 at commit [`4ae834b`](https://github.com/apache/spark/commit/4ae834b6c5aafc36d44adba4c4dbb934a63ea3ed). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58681651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21595/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18641539
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ // to generate a new url string scheme://server:port+path
--- End diff --
Hi @vanzin, actually here i just refer to the code of jetty 9
see(https://github.com/eclipse/jetty.project/blob/master/jetty-util/src/main/java/org/eclipse/jetty/util/URIUtil.java#L726-L733)
since there is no ```newURI```method in spark jetty version(spark use jetty
8).
And L238 is for the case to handle IPv6 address, here we can remove it if
unnecessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18532610
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ // to generate a new url string scheme://server:port+path
--- End diff --
Hi @scwf,
The reason I asked for a comment is that it seems like the method is doing
a little more than just that. For example, L238 seems to be doing some sort of
parsing of the `server` string, so it's more than just concatenating the
different arguments into a URL.
It would be nice if the comment explained exactly what the relationship
between the input and the output is. A unit test wouldn't hurt either.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18486828
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -180,12 +186,32 @@ private[spark] object JettyUtils extends Logging {
serverName: String = ): ServerInfo = {
val collection = new ContextHandlerCollection
-collection.setHandlers(handlers.toArray)
addFilters(handlers, conf)
// Bind to the given port, or throw a java.net.BindException if the
port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
+ val server = new Server
+ // Create a connector on port currentPort to listen for HTTP requests
+ val httpConnector = new SelectChannelConnector()
+ httpConnector.setPort(currentPort)
+ httpConnector.setHost(hostName)
+
+ if (conf.get(spark.ui.https.enabled, false).toBoolean) {
+// / If the new port wraps around, do not try a privilege port
+val securePort = (currentPort + 1 - 1024) % (65536 - 1024) + 1024
+val schema = https
--- End diff --
nit: scheme, not schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493363
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = new StringBuilder
+
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
builder.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ builder.append(scheme).append(://).append(server)
+}
+builder.append(':').append(port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
--- End diff --
override
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493360
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
--- End diff --
Can you add a comment explaining what this method is doing? It's not
obvious from reading the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493406
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = new StringBuilder
+
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
builder.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ builder.append(scheme).append(://).append(server)
+}
+builder.append(':').append(port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+target: String,
--- End diff --
nit: these lines look indented too much. They should be 4 spaces from where
the @Override currently is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493470
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = new StringBuilder
+
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
builder.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ builder.append(scheme).append(://).append(server)
+}
+builder.append(':').append(port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+target: String,
+baseRequest: Request,
+request: HttpServletRequest,
+response: HttpServletResponse): Unit = {
+if (baseRequest.isSecure) {
+ return
+}
+val url = newURI(schema, baseRequest.getServerName, securePort,
--- End diff --
nit: URL or URI?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493480
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = new StringBuilder
+
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
builder.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ builder.append(scheme).append(://).append(server)
+}
+builder.append(':').append(port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+target: String,
+baseRequest: Request,
+request: HttpServletRequest,
+response: HttpServletResponse): Unit = {
+if (baseRequest.isSecure) {
+ return
+}
+val url = newURI(schema, baseRequest.getServerName, securePort,
+ baseRequest.getRequestURI, baseRequest.getQueryString)
+response.setContentLength(0)
+response.sendRedirect(url)
--- End diff --
Might be a good idea to call ` response.encodeRedirectURL()` first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493516
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = new StringBuilder
+
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
builder.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ builder.append(scheme).append(://).append(server)
+}
+builder.append(':').append(port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+target: String,
+baseRequest: Request,
+request: HttpServletRequest,
+response: HttpServletResponse): Unit = {
+if (baseRequest.isSecure) {
+ return
+}
+val url = newURI(schema, baseRequest.getServerName, securePort,
+ baseRequest.getRequestURI, baseRequest.getQueryString)
+response.setContentLength(0)
+response.sendRedirect(url)
+baseRequest.setHandled(true)
+ }
+})
+redirectHandler
+ }
+
/** Attach a prefix to the given path, but avoid returning an empty path
*/
private def attachPrefix(basePath: String, relativePath: String): String
= {
if (basePath == ) relativePath else (basePath +
relativePath).stripSuffix(/)
}
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf):
Connector = {
+val ctxFactory = new SslContextFactory()
+conf.getAll
+ .filter { case (k, v) = k.startsWith(spark.ui.ssl.) }
+ .foreach { case (k, v) = setSslContextFactoryProps(k, v,
ctxFactory) }
+
+val connector = new SslSelectChannelConnector(ctxFactory)
+connector.setPort(port)
+connector
+ }
+
+ private def setSslContextFactoryProps(
+ key: String, value: String, ctxFactory: SslContextFactory) = {
+key match {
+ case spark.ui.ssl.client.https.needAuth =
ctxFactory.setNeedClientAuth(value.toBoolean)
+ case spark.ui.ssl.server.keystore.keypassword =
ctxFactory.setKeyManagerPassword(value)
+ case spark.ui.ssl.server.keystore.location =
ctxFactory.setKeyStorePath(value)
+ case spark.ui.ssl.server.keystore.password =
ctxFactory.setKeyStorePassword(value)
+ case spark.ui.ssl.server.keystore.type =
ctxFactory.setKeyStoreType(value)
+ case spark.ui.ssl.server.truststore.location =
ctxFactory.setTrustStore(value)
+ case spark.ui.ssl.server.truststore.password =
ctxFactory.setTrustStorePassword(value)
+ case spark.ui.ssl.server.truststore.type =
ctxFactory.setTrustStoreType(value)
+}
+
+ctxFactory
--- End diff --
Not needed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58118253 For some reason I downloaded the patch but it doesn't apply cleanly to my copy of the repo... I'll try again later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18497130
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = new StringBuilder
+
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
builder.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ builder.append(scheme).append(://).append(server)
+}
+builder.append(':').append(port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+target: String,
+baseRequest: Request,
+request: HttpServletRequest,
+response: HttpServletResponse): Unit = {
+if (baseRequest.isSecure) {
+ return
+}
+val url = newURI(schema, baseRequest.getServerName, securePort,
--- End diff --
```newURL``` will return a redirected https url string
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58128776 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21360/consoleFull) for PR 1980 at commit [`8b32853`](https://github.com/apache/spark/commit/8b32853f4d138027e0319196978d3de9bad32b1b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58132332 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21360/consoleFull) for PR 1980 at commit [`8b32853`](https://github.com/apache/spark/commit/8b32853f4d138027e0319196978d3de9bad32b1b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-58132338 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21360/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57905663 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21294/consoleFull) for PR 1980 at commit [`baaa1ce`](https://github.com/apache/spark/commit/baaa1ce05fcb426de7d3002a5cc30f18ae119d34). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57907416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21294/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57907414 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21294/consoleFull) for PR 1980 at commit [`baaa1ce`](https://github.com/apache/spark/commit/baaa1ce05fcb426de7d3002a5cc30f18ae119d34). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57909017 Updated and fix the confilct, @JoshRosen you can have a test for this refer to my last comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432489
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -180,12 +186,32 @@ private[spark] object JettyUtils extends Logging {
serverName: String = ): ServerInfo = {
val collection = new ContextHandlerCollection
-collection.setHandlers(handlers.toArray)
addFilters(handlers, conf)
// Bind to the given port, or throw a java.net.BindException if the
port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
+ val server = new Server
+ // Create a connector on port currentPort to listen for HTTP requests
+ val httpConnector = new SelectChannelConnector()
+ httpConnector.setPort(currentPort)
+ httpConnector.setHost(hostName)
+
+ if (conf.get(spark.ui.https.enabled, false).toBoolean) {
+// / If the new port wraps around, do not try a privilege port
+val securePort = (currentPort + 1 - 1024) % (65536 - 1024) + 1024
--- End diff --
What if this port isn't free? It's dangerous to make assumptions about
whether a particular ports is free, especially since this can cause problems in
Jenkins, where we might have multiple builds running concurrently on the same
machine that are contending for ports.
Could we use `Utils.startServiceOnPort` for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432532
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = newURIBuilder(scheme, server, port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+val builder = new StringBuilder
+appendSchemeHostPort(builder, scheme, server, port)
+builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String,
server: String, port: Int) {
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
url.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ url.append(scheme).append(://).append(server)
+}
+if (port 0) {
+ url.append(':').append(port)
+}
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+target: String,
+baseRequest: Request,
+request: HttpServletRequest,
+response: HttpServletResponse): Unit = {
+if (baseRequest.isSecure) {
+ return
+}
+if (securePort 0) {
+ val url = newURI(schema, baseRequest.getServerName, securePort,
+baseRequest.getRequestURI, baseRequest.getQueryString)
+ response.setContentLength(0)
+ response.sendRedirect(url)
+}else {
--- End diff --
Style nit: space after the `{`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432535
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = newURIBuilder(scheme, server, port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+val builder = new StringBuilder
+appendSchemeHostPort(builder, scheme, server, port)
+builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String,
server: String, port: Int) {
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
url.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ url.append(scheme).append(://).append(server)
+}
+if (port 0) {
+ url.append(':').append(port)
+}
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+target: String,
+baseRequest: Request,
+request: HttpServletRequest,
+response: HttpServletResponse): Unit = {
+if (baseRequest.isSecure) {
+ return
+}
+if (securePort 0) {
--- End diff --
Why would this port ever be negative or zero?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432538
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = newURIBuilder(scheme, server, port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
--- End diff --
I'd remove this function and fold it's functionality into the above
function, since it's only two lines of straightforward code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432542
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = newURIBuilder(scheme, server, port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+val builder = new StringBuilder
+appendSchemeHostPort(builder, scheme, server, port)
+builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String,
server: String, port: Int) {
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
url.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ url.append(scheme).append(://).append(server)
+}
+if (port 0) {
--- End diff --
I find this confusing: why would `port` ever be zero or negative?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432549
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = newURIBuilder(scheme, server, port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+val builder = new StringBuilder
+appendSchemeHostPort(builder, scheme, server, port)
+builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String,
server: String, port: Int) {
--- End diff --
Similarly, this function is only called from one place and it's kind of
confusing, so I'd prefer to inline this code in `newURI` (or define it as a
nested block / function there).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432553
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = newURIBuilder(scheme, server, port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+val builder = new StringBuilder
+appendSchemeHostPort(builder, scheme, server, port)
+builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String,
server: String, port: Int) {
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
url.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ url.append(scheme).append(://).append(server)
+}
+if (port 0) {
+ url.append(':').append(port)
+}
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+target: String,
+baseRequest: Request,
+request: HttpServletRequest,
+response: HttpServletResponse): Unit = {
+if (baseRequest.isSecure) {
+ return
+}
+if (securePort 0) {
+ val url = newURI(schema, baseRequest.getServerName, securePort,
+baseRequest.getRequestURI, baseRequest.getQueryString)
+ response.setContentLength(0)
+ response.sendRedirect(url)
+}else {
+ response.sendError(HttpStatus.FORBIDDEN_403, !Secure)
+}
+baseRequest.setHandled(true)
+ }
+})
+redirectHandler
+ }
+
/** Attach a prefix to the given path, but avoid returning an empty path
*/
private def attachPrefix(basePath: String, relativePath: String): String
= {
if (basePath == ) relativePath else (basePath +
relativePath).stripSuffix(/)
}
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf):
Connector = {
+val ctxFactory = new SslContextFactory()
+conf.getAll
+ .filter { case (k, v) = k.startsWith(spark.ui.ssl.) }
+ .foreach { case (k, v) = setSslContextFactoryProps(k, v,
ctxFactory) }
+
+val connector = new SslSelectChannelConnector(ctxFactory)
+connector.setPort(port)
+connector
+ }
+
+ private def setSslContextFactoryProps(
+ key: String, value: String, ctxFactory:SslContextFactory) = {
--- End diff --
Space after `:`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432556
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -180,12 +186,32 @@ private[spark] object JettyUtils extends Logging {
serverName: String = ): ServerInfo = {
val collection = new ContextHandlerCollection
-collection.setHandlers(handlers.toArray)
addFilters(handlers, conf)
// Bind to the given port, or throw a java.net.BindException if the
port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
+ val server = new Server
+ // Create a connector on port currentPort to listen for HTTP requests
+ val httpConnector = new SelectChannelConnector()
+ httpConnector.setPort(currentPort)
+ httpConnector.setHost(hostName)
+
+ if (conf.get(spark.ui.https.enabled, false).toBoolean) {
+// / If the new port wraps around, do not try a privilege port
+val securePort = (currentPort + 1 - 1024) % (65536 - 1024) + 1024
--- End diff --
Hey, note ```connect``` will be passed to ```Utils.startServiceOnPort```,
so in ```Utils.startServiceOnPort``` it will try to start http service on
currentPort and https service on securePort. If securePort is not free, it will
try again until the maximum number of retries
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432581
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -180,12 +186,32 @@ private[spark] object JettyUtils extends Logging {
serverName: String = ): ServerInfo = {
val collection = new ContextHandlerCollection
-collection.setHandlers(handlers.toArray)
addFilters(handlers, conf)
// Bind to the given port, or throw a java.net.BindException if the
port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
+ val server = new Server
+ // Create a connector on port currentPort to listen for HTTP requests
+ val httpConnector = new SelectChannelConnector()
+ httpConnector.setPort(currentPort)
+ httpConnector.setHost(hostName)
+
+ if (conf.get(spark.ui.https.enabled, false).toBoolean) {
+// / If the new port wraps around, do not try a privilege port
+val securePort = (currentPort + 1 - 1024) % (65536 - 1024) + 1024
--- End diff --
Ah, I see. Thanks for explaining.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432586
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path:
String, query: String) = {
+val builder = newURIBuilder(scheme, server, port)
+builder.append(path)
+if (query != null query.length 0)
builder.append('?').append(query)
+builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+val builder = new StringBuilder
+appendSchemeHostPort(builder, scheme, server, port)
+builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String,
server: String, port: Int) {
+if (server.indexOf(':') = 0 server.charAt(0) != '[') {
+
url.append(scheme).append(://).append('[').append(server).append(']')
+} else {
+ url.append(scheme).append(://).append(server)
+}
+if (port 0) {
+ url.append(':').append(port)
+}
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String):
ContextHandler = {
+val redirectHandler: ContextHandler = new ContextHandler
+redirectHandler.setContextPath(/)
+redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+target: String,
+baseRequest: Request,
+request: HttpServletRequest,
+response: HttpServletResponse): Unit = {
+if (baseRequest.isSecure) {
+ return
+}
+if (securePort 0) {
--- End diff --
Hmm, yes, it a no use condition :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57922537 Thanks for posting those configuration instructions. I think the problem was that I had been specifying an invalid truststore. After following your instructions, I was able to browse to the HTTP and HTTPs urls as expected and the redirect seemed to work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57922856 Updated, thanks for all your comments, it's very helpful --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57922888 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21303/consoleFull) for PR 1980 at commit [`a48c6fc`](https://github.com/apache/spark/commit/a48c6fc585c7fdb9c859b00f9b7d7d46836706b8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57923018 I'd like to try this out with view ACLs / authentication enabled to see how that interacts with HTTPs. Do you have a set of instructions that I can follow to test this out? It would be great to test this on YARN, since I think that's the typical environment where people would use view ACLs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57923191 I have not tested it on yarn and i think we can refer the docs to config ACLs / authentication. Yeah, this test is necessary and important, i will also test it :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57924008 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21303/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57924007 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21303/consoleFull) for PR 1980 at commit [`a48c6fc`](https://github.com/apache/spark/commit/a48c6fc585c7fdb9c859b00f9b7d7d46836706b8). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57787121 Hi, @JoshRosen, refer to http://wiki.eclipse.org/Jetty/Howto/Configure_SSL you can generating keys and certificates by openssl or keytool 1 Generating Keys and Certificates with OpenSSL keytool use command as follows openssl genrsa -des3 -out spark.key openssl req -new -x509 -key spark.key -out spark.crt openssl pkcs12 -inkey spark.key -in spark.crt -export -out spark.pkcs12 keytool -importkeystore -srckeystore spark.pkcs12 -srcstoretype PKCS12 -destkeystore spark.keystore 2 Generating Keys and Certificates with JDK keytool use keytool -keystore spark.keystore -alias spark -genkey -keyalg RSA This two way both ok in my machine and here is my configs in spark ```spark-env.sh``` export SPARK_MASTER_OPTS=-Dspark.ui.https.enabled=true -Dspark.ui.ssl.server.keystore.location=/home/wf/code/https/ssl/spark.keystore -Dspark.ui.ssl.server.keystore.keypassword=123456 -Dspark.ui.ssl.server.keystore.password=123456 export SPARK_WORKER_OPTS=-Dspark.ui.https.enabled=true -Dspark.ui.ssl.server.keystore.location=/home/wf/code/https/ssl/spark.keystore -Dspark.ui.ssl.server.keystore.keypassword=123456 -Dspark.ui.ssl.server.keystore.password=123456 ```spark-defaults.conf``` spark.ui.https.enabled true spark.ui.ssl.server.keystore.location /home/wf/code/https/ssl/spark.keystore spark.ui.ssl.server.keystore.keypassword 123456 spark.ui.ssl.server.keystore.password 123456 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57789135 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21244/consoleFull) for PR 1980 at commit [`e5c87cb`](https://github.com/apache/spark/commit/e5c87cb596ab3ae16ae52458c8e1edf6d1d3647a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57796385 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21244/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57796375 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21244/consoleFull) for PR 1980 at commit [`e5c87cb`](https://github.com/apache/spark/commit/e5c87cb596ab3ae16ae52458c8e1edf6d1d3647a). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57652480 @JoshRosen @andrewor14 @vanzin can you take a look at this to see whether it is ok to go~ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57725678 Have added redirect from http to https and updated docs. Some directions as follows 1 There is no changes for web ui when not set ```spark.ui.https.enabled``` 2 When https enabled, users no need input https url to get the right ui. Actually it's the same as http situation, users can input http url as they do in http case and the http url will redirect to a https one automatically 3 If set ```spark.ui.https.enabled```, user should(must) also config the ssl config opts spark.ui.ssl.server.keystore.keypassword spark.ui.ssl.server.keystore.location spark.ui.ssl.server.keystore.password. 4 Config ```spark.ui.ssl.client.https.needAuth``` is optional, used to enable client authenticates. When set true users should config client ssl config ```spark.ui.ssl.server.truststore.location``` and ```spark.ui.ssl.server.truststore.password```. 5 ```spark.ui.ssl.server.keystore.type``` and ```spark.ui.ssl.server.truststore.type``` has default value JKS --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57725962 Hey, sorry for the delay; I've been swamped with other work. I'm going to take a look right now... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57730551 I'm testing this out locally with my own keystore and truststore. Still debugging some issues (maybe I've misconfigured my keys), but I noticed that even with HTTPS enabled, the logs still refer to the `http://` URL: ``` 4/10/02 17:01:19 INFO Utils: start SparkUI at tryport: 4040 14/10/02 17:01:19 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 14/10/02 17:01:19 INFO Utils: start SparkUI at tryport: 4041 14/10/02 17:01:19 INFO Utils: Successfully started service 'SparkUI' on port 4041. 14/10/02 17:01:19 INFO SparkUI: Started SparkUI at http://joshs-mbp:4041 ``` I think this is because that log message is generated from https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/WebUI.scala#L103 which doesn't know about SSL. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57730857 Yes, here i did not change it, because http://joshs-mbp:4041; will redirect to https url --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57731094 The table in the documentation should include `spark.ui.https.enabled`, too; looks like it's missing now, which was confusing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57731222 Ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57734206 I followed the instructions at http://stackoverflow.com/a/14408381/590203 to generate a self-signed certificate, but unfortunately I see errors when I try to browse to my web UI: ``` javax.net.ssl.SSLHandshakeException: null cert chain at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1290) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513) at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1177) at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1149) at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) at org.eclipse.jetty.io.nio.SslConnection.wrap(SslConnection.java:460) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:386) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:193) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: javax.net.ssl.SSLHandshakeException: null cert chain at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:278) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:266) at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1631) at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:176) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868) at sun.security.ssl.Handshaker$1.run(Handshaker.java:808) at sun.security.ssl.Handshaker$1.run(Handshaker.java:806) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:375) ... 6 more ``` followed by ``` javax.net.ssl.SSLProtocolException: handshake alert: no_certificate at sun.security.ssl.ServerHandshaker.handshakeAlert(ServerHandshaker.java:1586) at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1747) at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1060) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:884) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:536) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:401) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:193) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) ``` I'm not too familiar with Java SSL setup; any idea what I'm doing wrong here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57468355 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21113/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57474002 Have added redirect from http to https and updated docs. Some directions as follows 1 there is no changes for web ui when not set ```spark.ui.https.enabled``` 2 when https enabled, users no need input https url to get the right ui. Aactually it's the same as http situation, users can input http url as they do in old case and the http will redirect to a https one automatically 3 if set ```spark.ui.https.enabled```, user should(must) also config the ssl config opts ```spark.ui.ssl.server.keystore.keypassword```, ```spark.ui.ssl.server.keystore.location``, ```spark.ui.ssl.server.keystore.password```. 4 config ```spark.ui.ssl.client.https.needAuth``` is optional, used to enable client authenticates. When set true users should config client ssl config ```spark.ui.ssl.server.truststore.location``` ```spark.ui.ssl.server.truststore.password```. 5 ```spark.ui.ssl.server.keystore.type``` and ```spark.ui.ssl.server.truststore.type``` has default value JKS --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57479925 jenkins down... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57480425 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21119/consoleFull) for PR 1980 at commit [`6c31dc7`](https://github.com/apache/spark/commit/6c31dc71a932083e158ef9c6737f33ebe9de1348). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57493293 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21119/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57493281 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21119/consoleFull) for PR 1980 at commit [`6c31dc7`](https://github.com/apache/spark/commit/6c31dc71a932083e158ef9c6737f33ebe9de1348). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57495153 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21129/consoleFull) for PR 1980 at commit [`7a898fb`](https://github.com/apache/spark/commit/7a898fbbb566172ee33f462357bb4fb02cced237). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57505455 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21129/consoleFull) for PR 1980 at commit [`7a898fb`](https://github.com/apache/spark/commit/7a898fbbb566172ee33f462357bb4fb02cced237). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57505464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21129/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57564475 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21165/consoleFull) for PR 1980 at commit [`a29ec86`](https://github.com/apache/spark/commit/a29ec8632cce8cb29c38d8e9e3aee2334400130b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57570330 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21165/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57570323 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21165/consoleFull) for PR 1980 at commit [`a29ec86`](https://github.com/apache/spark/commit/a29ec8632cce8cb29c38d8e9e3aee2334400130b). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` println(sFailed to load main class $childMainClass.)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17992049
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -183,7 +185,8 @@ private[spark] object JettyUtils extends Logging {
// Bind to the given port, or throw a java.net.BindException if the
port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
--- End diff --
It looks like we used to bind Jetty to a specific hostname / interface, but
it doesn't look like the new code uses `hostName` anywhere. To preserve the
same behavior, maybe we should call `connector.setHost()` after
`connector.setPort()` in the code below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17992230
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String
= {
if (basePath == ) relativePath else (basePath +
relativePath).stripSuffix(/)
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+val https = getHttpPolicy(conf)
+if (https) {
+ buildSslSelectChannelConnector(port, conf)
+} else {
+ conf.set(spark.http.policy, http)
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+}
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf):
Connector =
+ {
+val connector = new SslSelectChannelConnector
+connector.setPort(port)
+
+val context = connector.getSslContextFactory
+val needAuth = conf.getBoolean(spark.client.https.need-auth, false)
+
+context.setNeedClientAuth(needAuth)
+
context.setKeyManagerPassword(conf.get(spark.ssl.server.keystore.keypassword,
123456))
--- End diff --
It looks like we don't specify a default for
`spark.ssl.server.keystore.location` or for
`spark.ssl.server.truststore.password`, so we probably shouldn't set one for
`spark.ssl.server.keystore.keypassword` either. Maybe we should log a warning
if a user attempts to enable HTTPS without having properly configured their
keystore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56720228 Regarding `http - https` redirects, my PR #2474 adds Selenium-based tests to the Web UI; that should make it much easier to test this behavior. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18005297
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -137,6 +138,12 @@ private[spark] class Worker(
context.system.eventStream.subscribe(self,
classOf[RemotingLifecycleEvent])
webUi = new WorkerWebUI(this, workDir, webUiPort)
webUi.bind()
+val workerWebUiUrlPrefix = if( conf.get(spark.ui.https.enabled,
false).toBoolean) {
--- End diff --
nit: `if (conf...`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18005554
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String
= {
if (basePath == ) relativePath else (basePath +
relativePath).stripSuffix(/)
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+val https = getHttpPolicy(conf)
+if (https) {
+ buildSslSelectChannelConnector(port, conf)
+} else {
+ conf.set(spark.http.policy, http)
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+}
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf):
Connector =
+ {
+val connector = new SslSelectChannelConnector
+connector.setPort(port)
+
+val context = connector.getSslContextFactory
+val needAuth = conf.getBoolean(spark.client.https.need-auth, false)
+
+context.setNeedClientAuth(needAuth)
+
context.setKeyManagerPassword(conf.get(spark.ssl.server.keystore.keypassword,
123456))
--- End diff --
Something I'd recommend here instead is to just bypass options to Jetty.
e.g., some untested code based on similar Java code I have around:
val ctxFactory = new SslContextFactory();
conf.getAll.
.filter { case (k, v) = k.startsWith(spark.ui.ssl.) }
.foreach { case (k, v) = // set property in ctxFactory }
connector = new SslSelectChannelConnector(ctxFactory);
The trick is setting the property; in my code I use
`BeanUtils.setProperty()` for that. My main concern here is that Jetty's SSL
connector has a bunch of configurable options, and having a bypass mechanism
would avoid having to plumb all of them manually.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18005569
--- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala ---
@@ -92,6 +92,12 @@ private[spark] abstract class WebUI(
}
}
+ def appUiAddressPrefix = if(conf.get(spark.ui.https.enabled,
false).toBoolean) {
--- End diff --
nit: `if (`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56750923 `spark.ui.https.enabled` is repeated in several places; maybe a helper method somewhere to retrieve that property would be better? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18006403
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -183,7 +185,8 @@ private[spark] object JettyUtils extends Logging {
// Bind to the given port, or throw a java.net.BindException if the
port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
--- End diff --
here should only ```setHost()```, port has been set in
```getConnector(currentPort, conf)```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18006638
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String
= {
if (basePath == ) relativePath else (basePath +
relativePath).stripSuffix(/)
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+val https = getHttpPolicy(conf)
+if (https) {
+ buildSslSelectChannelConnector(port, conf)
+} else {
+ conf.set(spark.http.policy, http)
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+}
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf):
Connector =
+ {
+val connector = new SslSelectChannelConnector
+connector.setPort(port)
+
+val context = connector.getSslContextFactory
+val needAuth = conf.getBoolean(spark.client.https.need-auth, false)
+
+context.setNeedClientAuth(needAuth)
+
context.setKeyManagerPassword(conf.get(spark.ssl.server.keystore.keypassword,
123456))
--- End diff --
thanks, pretty good advice
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56301104 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20622/consoleFull) for PR 1980 at commit [`c90d84e`](https://github.com/apache/spark/commit/c90d84e0c70a88742feb453c6f1517163878c003). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56301129 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20622/consoleFull) for PR 1980 at commit [`c90d84e`](https://github.com/apache/spark/commit/c90d84e0c70a88742feb453c6f1517163878c003). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56301209 @JoshRosen, here i named the configuration options refer to how hadoop do(actually just use ```spark``` instead ```hadoop```). ```spark.client``` is the browser which connect to UIServer. Now browse to ```http```ui when configure ```http``` will get error, and i will check that how to set up automatic redirect(maybe configure this in jetty ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56301425 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20623/consoleFull) for PR 1980 at commit [`de8d1bd`](https://github.com/apache/spark/commit/de8d1bd5cc3c840230d6d5dd4a4866bf0f0f7575). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56302384 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20626/consoleFull) for PR 1980 at commit [`35074fd`](https://github.com/apache/spark/commit/35074fd465dee6c3dae8e0a43f2787aed50f4537). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56302482 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20623/consoleFull) for PR 1980 at commit [`de8d1bd`](https://github.com/apache/spark/commit/de8d1bd5cc3c840230d6d5dd4a4866bf0f0f7575). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56303535 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20626/consoleFull) for PR 1980 at commit [`35074fd`](https://github.com/apache/spark/commit/35074fd465dee6c3dae8e0a43f2787aed50f4537). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56303934 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20627/consoleFull) for PR 1980 at commit [`967950f`](https://github.com/apache/spark/commit/967950f26d612383fd3e121af3284d4a2ce013c8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56304208 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20628/consoleFull) for PR 1980 at commit [`9591c9c`](https://github.com/apache/spark/commit/9591c9c5b90e0d1d65b4a48b08b53f6a3ed3e49d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56304928 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20627/consoleFull) for PR 1980 at commit [`967950f`](https://github.com/apache/spark/commit/967950f26d612383fd3e121af3284d4a2ce013c8). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-56306346 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20628/consoleFull) for PR 1980 at commit [`9591c9c`](https://github.com/apache/spark/commit/9591c9c5b90e0d1d65b4a48b08b53f6a3ed3e49d). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/1980#discussion_r17569562 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala --- @@ -30,7 +30,7 @@ private[spark] class WorkerInfo( val cores: Int, val memory: Int, val actor: ActorRef, -val webUiPort: Int, +val webUiAddress: String, val publicAddress: String) --- End diff -- It looks like this `publicAddress` field is never read anymore, so we should remove it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/1980#discussion_r17569719 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala --- @@ -78,6 +78,7 @@ private[spark] class Worker( var activeMasterUrl: String = var activeMasterWebUiUrl : String = val akkaUrl = akka.tcp://%s@%s:%s/user/%s.format(actorSystemName, host, port, actorName) + var workerWebUiUrl: String = _ --- End diff -- For stylistic consistency with the surrounding code, I'd prefer if this was initialized with an empty string. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17569799
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -35,6 +35,8 @@ import org.json4s.jackson.JsonMethods.{pretty, render}
import org.apache.spark.{Logging, SecurityManager, SparkConf}
import org.apache.spark.util.Utils
+import org.eclipse.jetty.server.nio.SelectChannelConnector
--- End diff --
These imports should be placed alongside the other `jetty` imports (e.g.
line 32).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17569959
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String
= {
if (basePath == ) relativePath else (basePath +
relativePath).stripSuffix(/)
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+val https = getHttpPolicy(conf)
+if (https) {
+ buildSslSelectChannelConnector(port, conf)
+} else {
+ conf.set(spark.http.policy, http)
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+}
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf):
Connector =
+ {
+val connector = new SslSelectChannelConnector
+connector.setPort(port)
+
+val context = connector.getSslContextFactory
+val needAuth = conf.getBoolean(spark.client.https.need-auth, false)
+
+context.setNeedClientAuth(needAuth)
+
context.setKeyManagerPassword(conf.get(spark.ssl.server.keystore.keypassword,
123456))
+context.setKeyStorePath(conf.get(spark.ssl.server.keystore.location))
+
context.setKeyStorePassword(conf.get(spark.ssl.server.keystore.password,
123456))
+context.setKeyStoreType(conf.get(spark.ssl.server.keystore.type,
jks))
+
+if (needAuth conf.contains(spark.ssl.server.truststore.location))
{
+
context.setTrustStore(conf.get(spark.ssl.server.truststore.location))
+
context.setTrustStorePassword(conf.get(spark.ssl.server.truststore.password))
+
context.setTrustStoreType(conf.get(spark.ssl.server.truststore.type, jks))
+}
+connector
+ }
+
+ def getHttpPolicy(conf: SparkConf): Boolean = {
--- End diff --
This is a confusing name, since this function returns a boolean but sounds
like it will return a http policy. `isHttpsEnabled` would be a better name,
although I'm not sure that we need this:
We should just be able to do
```
val https = conf.get(spark.http.policy, ) == https
```
instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17570027
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String
= {
if (basePath == ) relativePath else (basePath +
relativePath).stripSuffix(/)
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+val https = getHttpPolicy(conf)
+if (https) {
+ buildSslSelectChannelConnector(port, conf)
+} else {
+ conf.set(spark.http.policy, http)
--- End diff --
Mutating the configuration object seems like an anti-pattern to me; can we
avoid this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17570077
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String
= {
if (basePath == ) relativePath else (basePath +
relativePath).stripSuffix(/)
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+val https = getHttpPolicy(conf)
+if (https) {
+ buildSslSelectChannelConnector(port, conf)
+} else {
+ conf.set(spark.http.policy, http)
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+}
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf):
Connector =
+ {
+val connector = new SslSelectChannelConnector
+connector.setPort(port)
+
+val context = connector.getSslContextFactory
+val needAuth = conf.getBoolean(spark.client.https.need-auth, false)
+
+context.setNeedClientAuth(needAuth)
+
context.setKeyManagerPassword(conf.get(spark.ssl.server.keystore.keypassword,
123456))
--- End diff --
Shouldn't this be configurable instead of hardcoded?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
