Wang Yuan created SPARK-35229:
---------------------------------
Summary: Spark Job web page is extremely slow while there are more
than 1500 events in timeline
Key: SPARK-35229
URL: https://issues.apache.org/jira/browse/SPARK-35229
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.1.1
Reporter: Wang Yuan
In a spark streaming application, there are 1000+ executors, more than 2000
events (executors events, job events) generated, then the jobs/job web page of
spark is not able to show. The browser (chrome, firefox, safari) freezes. I had
to open another window to open stages, executors pages by their addresses
manually.
The jobs page is the home page, so this is rather annoyed. The problem is that
the vis-timeline rending is too slow.
Their are some suggestion:
1) the page should not render timeline when the page is loading unless user
clicks the link.
2) the executor group and job group should be separated, and user can choose to
show one, both or neither.
3) the executor group should display executor event by time horizontally.
Currently it is displayed executors one line by one line. If more than 100
executors, the page is not that good.
4)the vis-timeline library is not maintained anymore since 2017. Should be
replaced a new one like [https://github.com/visjs/vis-timeline]
5) it is also good to get recent events, e.g. 500, and load more if user want
see more, however the data could all be loaded once.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
