Ye Zhou created SPARK-23608:
-------------------------------
Summary: SHS needs synchronization between attachSparkUI and
detachSparkUI functions
Key: SPARK-23608
URL: https://issues.apache.org/jira/browse/SPARK-23608
Project: Spark
Issue Type: Bug
Components: Spark Core, Web UI
Affects Versions: 2.3.0
Reporter: Ye Zhou
We continuously hit an issue with SHS after it runs for a while and have some
REST API calls to it. SHS suddenly shows an empty home page with 0 application.
It is caused by the unexpected JSON data returned from rest call
"api/v1/applications?limit=8000". This REST call returns the home page html
codes instead of list of application summary. Some other REST call which asks
for application detailed information also returns home page html codes. But
there are still some working REST calls. We have to restart SHS to solve the
issue.
We attached remote debugger to the problematic process and checked the attached
jetty handlers tree in the web server. We found that the jetty handler added by
"attachHandler(ApiRootResource.getServletHandler(this))" is not in the tree as
well as some other handlers. Without the root resource servlet handler, SHS
will not work correctly serving both UI and REST calls. SHS will directly
return the HistoryServerPage html to user as it cannot find handlers to handle
the request.
Spark History Server has to attachSparkUI in order to serve user requests. The
application SparkUI getting attached when the application details data gets
loaded into Guava Cache. While attaching SparkUI, SHS will add attach all jetty
handlers into the current web service. But while the data gets cleared out from
Guava Cache, SHS will detach all the application's SparkUI jetty handlers. Due
to the asynchronous feature in Guava Cache, the clear out from cache is not
synchronized with loading into cache. The actual clear out in Guava Cache which
triggers detachSparkUI might be detaching the handlers while the attachSparkUI
is attaching jetty handlers.
After adding synchronization between attachSparkUI and detachSparkUI in history
server, this issue never happens again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
