Matt Cheah created SPARK-4679: --------------------------------- Summary: Race condition in querying the Spark UI JSON endpoint when Jetty context handlers are added and removed Key: SPARK-4679 URL: https://issues.apache.org/jira/browse/SPARK-4679 Project: Spark Issue Type: Bug Affects Versions: 1.0.2 Reporter: Matt Cheah
We started seeing some strange behavior when we were querying the Spark UI JSON endpoint for job metadata. When the Spark cluster was under heavy load from a large number of short-lived spark contexts being created and stopped, querying the JSON endpoint (e.g. https://localhost:8080/json) returned the HTML webpage instead. We were relying on this JSON data to get information about running jobs on our own server and the result was a JSON Parse Exception. I dug into the code and realized that this is caused by a race condition between how we add and remove Jetty context handlers on the Akka message queue thread and how the context handler is looked up on a different thread when the HTTP request is fired. Whenever an application is started or completes, we invoke ContextHandlerCollection.setHandlers() adding or removing a new Jetty handler to the collection. However, setHandlers() first sets its internal collection to null before configuring the new passed-in collection. If an HTTP request is made and the Jetty context handler is looked up AFTER the collection's internal map is set to null, but BEFORE it has configured the new collection, the default handler is selected to return HTML. tl;dr we're using Jetty's ContextHandlerCollection in a way that is not thread-safe. The issue we found is only one possible ramification of this; I'm not sure what other consequences a non-thread-safe usage of Jetty may have. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org