[
https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15689429#comment-15689429
]
Apache Spark commented on SPARK-17843:
--------------------------------------
User 'vijoshi' has created a pull request for this issue:
https://github.com/apache/spark/pull/15991
> History Server gives no feedback about the application list being incomplete
> ----------------------------------------------------------------------------
>
> Key: SPARK-17843
> URL: https://issues.apache.org/jira/browse/SPARK-17843
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 2.0.0, 2.0.1
> Reporter: Vinayak Joshi
> Assignee: Vinayak Joshi
> Priority: Minor
> Fix For: 2.1.0, 2.2.0
>
>
> A user who lands on the history server's application listing page before the
> history server's background processing has finished replaying the existing
> event logs complains why a particular application is not included in the
> list. There is no indication on the UI about the possible reason, or that
> refreshing the page after some time may show up the application the user
> expects to see. This problem is very noticeable when there are large sized
> event logs that take a long time to finish replaying.
> The usability problems with large (number or size) event logs and the history
> server have been known. Particularly SPARK-5522, SPARK-13988 and other issues
> referenced within them highlight the problems and the improvements done so
> far. To improve the history server startup time and reduce the impact of
> large event log files, the event logs are now processed (replayed) via a pool
> of threads. This allows a user to browse to the application listing page
> before the event logs have finished replaying. After history server startup,
> a user expects to see any old completed applications to appear in the
> application list page. But unless the corresponding event logs have finished
> replaying, the application wont be in the list, and the user complains. There
> is no feedback to the user about this on the UI, hence this JIRA to try and
> address this problem.
> Idea is to give some indication of the number of event logs that are pending
> replay to the user. Note that the way the replay is currently designed, one
> cycle of "check for logs that need to be replayed > replay the logs > update
> application info" needs to complete before a new one begins. Therefore, it
> should be possible for the FsApplicationHistoryProvider to send info about
> the number of logs that are currently pending processing. This in turn would
> address the user anxiety of not seeing the application they expect to see.
> I will be attaching a pull request with my initial take on implementing this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
