Shixiong Zhu created SPARK-27394:
------------------------------------
Summary: The staleness of UI may last minutes or hours when no
tasks start or finish
Key: SPARK-27394
URL: https://issues.apache.org/jira/browse/SPARK-27394
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 2.4.1, 2.4.0
Reporter: Shixiong Zhu
Run the following codes on a cluster that has at least 2 cores.
{code}
sc.makeRDD(1 to 1000, 1000).foreach { i =>
Thread.sleep(300000)
}
{code}
The jobs page will just show one running task.
This is because when the second task event calls
"AppStatusListener.maybeUpdate" for a job, it will just ignore since the gap
between two events is smaller than `spark.ui.liveUpdate.period`.
After the second task event, in the above case, because there won't be any
other task events, the Spark UI will be always stale until the next task event
gets fired (after 300 seconds).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
