Hi There, We're currently using HDP 2.3.4, Spark 1.5.2 with a Spark Streaming job in YARN Cluster mode consuming from a high volume Kafka topic. When we try to access the Spark Streaming UI on the application master, it is unresponsive/hangs or sometimes comes back with connection refused.
It seems this UI is resident on the driver, and looking at its thread dump we see the below. Other tabs in the UI are fine. Does anyone have any ideas? Any further info required just ask. Thread 258: qtp1595613401-258 - /streaming/ (BLOCKED) org.apache.spark.streaming.DStreamGraph.getInputStreamName(DStreamGraph.scala:114) org.apache.spark.streaming.ui.StreamingJobProgressListener.streamName(StreamingJobProgressListener.scala:188) org.apache.spark.streaming.ui.StreamingPage$$anonfun$21.apply(StreamingPage.scala:429) org.apache.spark.streaming.ui.StreamingPage$$anonfun$21.apply(StreamingPage.scala:429) scala.Option.orElse(Option.scala:257) org.apache.spark.streaming.ui.StreamingPage.org$apache$spark$streaming$ui$StreamingPage$$generateInputDStreamRow(StreamingPage.scala:429) org.apache.spark.streaming.ui.StreamingPage$$anonfun$18.apply(StreamingPage.scala:396) org.apache.spark.streaming.ui.StreamingPage$$anonfun$18.apply(StreamingPage.scala:395) scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) scala.collection.TraversableLike$class.map(TraversableLike.scala:244) scala.collection.AbstractTraversable.map(Traversable.scala:105) org.apache.spark.streaming.ui.StreamingPage.generateInputDStreamsTable(StreamingPage.scala:395) org.apache.spark.streaming.ui.StreamingPage.generateStatTable(StreamingPage.scala:348) org.apache.spark.streaming.ui.StreamingPage.render(StreamingPage.scala:157) org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:79) org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:79) org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:69) javax.servlet.http.HttpServlet.service(HttpServlet.java:735) javax.servlet.http.HttpServlet.service(HttpServlet.java:848) org.spark-project.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) org.spark-project.jetty.server.Server.handle(Server.java:366) org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644) org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) java.lang.Thread.run(Thread.java:745) Thread 33: SparkListenerBus (BLOCKED) org.apache.spark.streaming.ui.StreamingJobProgressListener.onJobStart(StreamingJobProgressListener.scala:123) org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:34) org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136) org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Thread 541: streaming-job-executor-0 (WAITING) Thread 131: StreamingListenerBus (BLOCKED) org.apache.spark.streaming.ui.StreamingJobProgressListener.onBatchSubmitted(StreamingJobProgressListener.scala:93) org.apache.spark.streaming.scheduler.StreamingListenerBus.onPostEvent(StreamingListenerBus.scala:41) org.apache.spark.streaming.scheduler.StreamingListenerBus.onPostEvent(StreamingListenerBus.scala:26) org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136) org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Cheers, Tom Ellis Consultant Developer - Excelian Data Lake | Financial Markets IT LLOYDS BANK COMMERCIAL BANKING ________________________________ E: tom.el...@lloydsbanking.com<mailto:tom.el...@lloydsbanking.com> Website: www.lloydsbankcommercial.com<http://www.lloydsbankcommercial.com/> , , , Reduce printing. Lloyds Banking Group is helping to build the low carbon economy. Corporate Responsibility Report: www.lloydsbankinggroup-cr.com/downloads<http://www.lloydsbankinggroup-cr.com/downloads> Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. Telephone: 0345 603 1637 Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority. Cheltenham & Gloucester plc is authorised and regulated by the Financial Conduct Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813. This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.
