[
https://issues.apache.org/jira/browse/SPARK-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-20656:
------------------------------------
Assignee: (was: Apache Spark)
> Incremental parsing of event logs in SHS
> ----------------------------------------
>
> Key: SPARK-20656
> URL: https://issues.apache.org/jira/browse/SPARK-20656
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: Marcelo Masiero Vanzin
> Priority: Major
>
> This feature is mentioned in the spec attached to SPARK-18085 but there's not
> a lot of discussion about it.
> It would be good to implement incremental parsing of event logs in the SHS.
> With the new work, UI data is stored on disk, so it should be possible to
> save enough metadata about the event log and the state of the listeners to
> allow one to resume parsing the log of a live application at the point where
> it stopped in the previous iteration.
> This would considerably speed up parsing on updates, and could be done
> speculatively so that UIs for new applications are available in the SHS
> almost immediately.
> I'm filing this as a separate enhancement because I don't want to block
> SPARK-18085 on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
