Zoltán Zvara created SPARK-24281:
------------------------------------
Summary: Option to disable Spark UI's web filter in YARN-client
mode
Key: SPARK-24281
URL: https://issues.apache.org/jira/browse/SPARK-24281
Project: Spark
Issue Type: Improvement
Components: Web UI, YARN
Affects Versions: 2.3.0, 2.2.1
Reporter: Zoltán Zvara
When Spark is deployed to YARN (either in client or cluster mode), the driver's
web UI is accessible solely through YARN's web proxy due to security reasons.
The YARN web proxy usually started on one of the {{ResourceManager}}s (RM) of
YARN. A common use case for application developers is to deploy applications on
the cluster directly from their computer in client mode. This ensures fast
experiment deployments to the target cluster, while allows IDE-driven
breakpoints on the driver, which is kept on the user's computer; in addition to
keep all features that an IDE provides. In these setups, the user might be
connected through a VPN or networks where packets that initiate connection
might not find a route back to the driver. One example is when the user
attempts to access the web UI running on their computer. The web UI will
redirect to the proxy, which then attempts to access the web UI again. In this
process, the proxy might fail to serve pages from the web UI: no response or
redirect-loops can occur.
*This improvement allows users in YARN-client mode to access the driver's web
UI directly, bypassing the YARN proxy altogether.*
In the event of the {{ApplicationMaster}} is started, a web UI filter will be
applied on the endpoint, which then makes the web UI to redirect all calls not
addressed from any of the registered RMs.
An option to disable the web UI filters would help users in the aforementioned
scenarios to access the web UI from their local computer. An alternative is to
keep their pre-registered filters even if the YARN web UI filters are applied -
which now simply overwrites all user-specified filters.
A simple code to mitigate this problem and to make the web UI accessible
locally in a YARN-client scenario is the following:
{code:java}
context.ui.foreach {
_.getHandlers.asInstanceOf[Seq[ServletContextHandler]].foreach {
h =>
val filters = h.getServletHandler.getFilters
if (filters.nonEmpty) {
filters.foreach {
_.doStop()
}
h.getServletHandler.setFilters(null)
h.getServletHandler.setFilterMappings(null)
System.setProperty("spark.ui.proxyBase", "")
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
