Github user IngoSchuster commented on a diff in the pull request:
https://github.com/apache/spark/pull/18499#discussion_r125665745
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -194,30 +194,26 @@ private[spark] object JettyUtils extends Logging {
}
/** Create a handler for proxying request to Workers and Application
Drivers */
- def createProxyHandler(
- prefix: String,
- target: String): ServletContextHandler = {
+ def createProxyHandler(idToUiAddress: String => Option[String]):
ServletContextHandler = {
val servlet = new ProxyServlet {
override def rewriteTarget(request: HttpServletRequest): String = {
- val rewrittenURI = createProxyURI(
- prefix, target, request.getRequestURI(),
request.getQueryString())
- if (rewrittenURI == null) {
- return null
- }
- if (!validateDestination(rewrittenURI.getHost(),
rewrittenURI.getPort())) {
- return null
- }
- rewrittenURI.toString()
+ val path = request.getPathInfo
+ if (path == null) return null
+
+ val prefixTrailingSlashIndex = path.indexOf('/', 1)
+ val prefix = path.substring(0,
+ if (prefixTrailingSlashIndex == -1) path.length else
prefixTrailingSlashIndex)
+ val id = prefix.drop(1)
+
+ idToUiAddress(id)
+ .map(createProxyURI(prefix, _, path, request.getQueryString))
+ .filter(uri => uri != null && validateDestination(uri.getHost,
uri.getPort))
+ .map(_.toString)
+ .orNull
}
override def newHttpClient(): HttpClient = {
- // SPARK-21176: Use the Jetty logic to calculate the number of
selector threads (#CPUs/2),
- // but limit it to 8 max.
- // Otherwise, it might happen that we exhaust the threadpool since
in reverse proxy mode
- // a proxy is instantiated for each executor. If the head node has
many processors, this
- // can quickly add up to an unreasonably high number of threads.
- val numSelectors = math.max(1, math.min(8,
Runtime.getRuntime().availableProcessors() / 2))
- new HttpClient(new HttpClientTransportOverHTTP(numSelectors), null)
+ new HttpClient(new HttpClientTransportOverHTTP(1), null)
--- End diff --
I fully agree @aosagie - it's much better if we don't scale the number of
selector threads up with the number of executors.
But since we don't serve simple html pages but also page(elements) that
might have a higher latency than simply serving static files, I think we should
not risk that the page is build up from serial requests where latency adds up -
my server for example has 170ms network latency.
I suggest that we set the number of selector threads to 8. This is
certainly not excessive but still allows to serve a page with several elements
in parallel.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
