[
https://issues.apache.org/jira/browse/SPARK-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113670#comment-16113670
]
Josh Bacon commented on SPARK-20853:
------------------------------------
For the record, I'm experiencing the exact same behavior as described by
[~tmckay]. If total number of workers + drivers exceed 9 (each with
spark.ui.reverseProxy enabled), then the Master U.I. becomes unresponsive.
Remove either workers or running drivers below the threshold, the Master U.I.
will become responsive again.
> spark.ui.reverseProxy=true leads to hanging communication to master
> -------------------------------------------------------------------
>
> Key: SPARK-20853
> URL: https://issues.apache.org/jira/browse/SPARK-20853
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.1.0
> Environment: ppc64le GNU/Linux, POWER8, only master node is reachable
> externally other nodes are in an internal network
> Reporter: Benno Staebler
> Labels: network, web-ui
>
> When *reverse proxy is enabled*
> {quote}
> spark.ui.reverseProxy=true
> spark.ui.reverseProxyUrl=/
> {quote}
> first of all any invocation of the spark master Web UI hangs forever locally
> (e.g. http://192.168.10.16:25001) and via external URL without any data
> received.
> One, sometimes two spark applications succeed without error and than workers
> start throwing exceptions:
> {quote}
> Caused by: java.io.IOException: Failed to connect to /192.168.10.16:25050
> {quote}
> The application dies during creation of SparkContext:
> {quote}
> 2017-05-22 16:11:23 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting
> to master spark://node0101:25000...
> 2017-05-22 16:11:23 INFO TransportClientFactory:254 - Successfully created
> connection to node0101/192.168.10.16:25000 after 169 ms (132 ms spent in
> bootstraps)
> 2017-05-22 16:11:43 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting
> to master spark://node0101:25000...
> 2017-05-22 16:12:03 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting
> to master spark://node0101:25000...
> 2017-05-22 16:12:23 ERROR StandaloneSchedulerBackend:70 - Application has
> been killed. Reason: All masters are unresponsive! Giving up.
> 2017-05-22 16:12:23 WARN StandaloneSchedulerBackend:66 - Application ID is
> not initialized yet.
> 2017-05-22 16:12:23 INFO Utils:54 - Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 25056.
> .....
> Caused by: java.lang.IllegalArgumentException: requirement failed: Can only
> call getServletHandlers on a running MetricsSystem
> {quote}
> *This definitively does not happen without reverse proxy enabled!*
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
