Matt Cheah created SPARK-4679:
---------------------------------
Summary: Race condition in querying the Spark UI JSON endpoint
when Jetty context handlers are added and removed
Key: SPARK-4679
URL: https://issues.apache.org/jira/browse/SPARK-4679
Project: Spark
Issue Type: Bug
Affects Versions: 1.0.2
Reporter: Matt Cheah
We started seeing some strange behavior when we were querying the Spark UI JSON
endpoint for job metadata.
When the Spark cluster was under heavy load from a large number of short-lived
spark contexts being created and stopped, querying the JSON endpoint (e.g.
https://localhost:8080/json) returned the HTML webpage instead. We were relying
on this JSON data to get information about running jobs on our own server and
the result was a JSON Parse Exception.
I dug into the code and realized that this is caused by a race condition
between how we add and remove Jetty context handlers on the Akka message queue
thread and how the context handler is looked up on a different thread when the
HTTP request is fired. Whenever an application is started or completes, we
invoke ContextHandlerCollection.setHandlers() adding or removing a new Jetty
handler to the collection. However, setHandlers() first sets its internal
collection to null before configuring the new passed-in collection. If an HTTP
request is made and the Jetty context handler is looked up AFTER the
collection's internal map is set to null, but BEFORE it has configured the new
collection, the default handler is selected to return HTML.
tl;dr we're using Jetty's ContextHandlerCollection in a way that is not
thread-safe. The issue we found is only one possible ramification of this; I'm
not sure what other consequences a non-thread-safe usage of Jetty may have.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
