Hey Tom, Could you provide all blocked threads? Perhaps due to some potential deadlock.
On Fri, Jul 8, 2016 at 10:30 AM, Ellis, Tom (Financial Markets IT) < tom.el...@lloydsbanking.com.invalid> wrote: > Hi There, > > > > We’re currently using HDP 2.3.4, Spark 1.5.2 with a Spark Streaming job in > YARN Cluster mode consuming from a high volume Kafka topic. When we try to > access the Spark Streaming UI on the application master, it is > unresponsive/hangs or sometimes comes back with connection refused. > > > > It seems this UI is resident on the driver, and looking at its thread dump > we see the below. Other tabs in the UI are fine. Does anyone have any > ideas? Any further info required just ask. > > > > Thread 258: qtp1595613401-258 - /streaming/ (BLOCKED) > > > org.apache.spark.streaming.DStreamGraph.getInputStreamName(DStreamGraph.scala:114) > > > org.apache.spark.streaming.ui.StreamingJobProgressListener.streamName(StreamingJobProgressListener.scala:188) > > > org.apache.spark.streaming.ui.StreamingPage$$anonfun$21.apply(StreamingPage.scala:429) > > > org.apache.spark.streaming.ui.StreamingPage$$anonfun$21.apply(StreamingPage.scala:429) > > scala.Option.orElse(Option.scala:257) > > org.apache.spark.streaming.ui.StreamingPage.org > $apache$spark$streaming$ui$StreamingPage$$generateInputDStreamRow(StreamingPage.scala:429) > > > org.apache.spark.streaming.ui.StreamingPage$$anonfun$18.apply(StreamingPage.scala:396) > > > org.apache.spark.streaming.ui.StreamingPage$$anonfun$18.apply(StreamingPage.scala:395) > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) > > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > > scala.collection.AbstractTraversable.map(Traversable.scala:105) > > > org.apache.spark.streaming.ui.StreamingPage.generateInputDStreamsTable(StreamingPage.scala:395) > > > org.apache.spark.streaming.ui.StreamingPage.generateStatTable(StreamingPage.scala:348) > > org.apache.spark.streaming.ui.StreamingPage.render(StreamingPage.scala:157) > > org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:79) > > org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:79) > > org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:69) > > javax.servlet.http.HttpServlet.service(HttpServlet.java:735) > > javax.servlet.http.HttpServlet.service(HttpServlet.java:848) > > > org.spark-project.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) > > > org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) > > > org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > > > org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) > > > org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > > > org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > > > org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > > > org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > > org.spark-project.jetty.server.Server.handle(Server.java:366) > > > org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) > > > org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) > > > org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) > > org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644) > > org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > > > org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > > > org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) > > > org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > > > org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > > > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > > java.lang.Thread.run(Thread.java:745) > > > > Thread 33: SparkListenerBus (BLOCKED) > > > org.apache.spark.streaming.ui.StreamingJobProgressListener.onJobStart(StreamingJobProgressListener.scala:123) > > > org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:34) > > > org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) > > > org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) > > org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) > > > org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) > > > org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) > > org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136) > > > org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) > > Thread 541: streaming-job-executor-0 (WAITING) > > Thread 131: StreamingListenerBus (BLOCKED) > > > org.apache.spark.streaming.ui.StreamingJobProgressListener.onBatchSubmitted(StreamingJobProgressListener.scala:93) > > > org.apache.spark.streaming.scheduler.StreamingListenerBus.onPostEvent(StreamingListenerBus.scala:41) > > > org.apache.spark.streaming.scheduler.StreamingListenerBus.onPostEvent(StreamingListenerBus.scala:26) > > org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) > > > org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) > > > org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) > > org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136) > > > org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) > > > > > > > > > > *Cheers,* > > > > *Tom Ellis* > Consultant Developer – Excelian > > Data Lake | Financial Markets IT > *LLOYDS BANK COMMERCIAL BANKING* > ------------------------------ > > > *E: *tom.el...@lloydsbanking.com > Website: www.lloydsbankcommercial.com > , , , > Reduce printing. Lloyds Banking Group is helping to build the low carbon > economy. > Corporate Responsibility Report: www.lloydsbankinggroup-cr.com/downloads > > > > Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. > Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank > plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in > England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. > Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. > SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered > Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales > 2299428. Telephone: 0345 603 1637 > > Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential > Regulation Authority and regulated by the Financial Conduct Authority and > Prudential Regulation Authority. > > Cheltenham & Gloucester plc is authorised and regulated by the Financial > Conduct Authority. > > Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester > Savings is a division of Lloyds Bank plc. > > HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in > Scotland no. SC218813. > > This e-mail (including any attachments) is private and confidential and > may contain privileged material. If you have received this e-mail in error, > please notify the sender and delete it (including any attachments) > immediately. You must not copy, distribute, disclose or use any of the > information in it or any attachments. Telephone calls may be monitored or > recorded. >
