Andrei Taleanu created SPARK-20321:
--------------------------------------
Summary: Spark UI cannot be shutdown in spark streaming app
Key: SPARK-20321
URL: https://issues.apache.org/jira/browse/SPARK-20321
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.1.0
Reporter: Andrei Taleanu
When an exception thrown the transform stage is handled in foreachRDD and the
streaming context is forced to stop, the SparkUI appears to hang and
continually dump the following logs in an infinite loop:
{noformat}
...
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from
select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on
select
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from
select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on
select
...
{noformat}
Unfortunately I don't have a minimal example that reproduces this issue but
here is what I can share:
{noformat}
val dstream = pull data from kafka
val mapped = dstream transform { rdd =>
val data = getData // Perform a call that potentially throws an exception
// broadcast the data
// flatMap the RDD using the data
}
mapped.foreachRDD {
try {
// write some data in a DB
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
mapped.foreachRDD {
try {
// write data to Kafka
// manually checkpoint the Kafka offsets (because I need them in JSON
format)
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
{noformat}
The issue appears when stop is invoked. At the point when SparkUI is stopped,
it enters that infinite loop. Initially I thought it relates to Jetty, as the
version used in SparkUI had some bugs (e.g. [this
one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped Jetty to a
more recent version (March 2017) but still got the error.
I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of Mesos.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
