Marcelo Vanzin created SPARK-20656:
--------------------------------------
Summary: Incremental parsing of event logs in SHS
Key: SPARK-20656
URL: https://issues.apache.org/jira/browse/SPARK-20656
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.3.0
Reporter: Marcelo Vanzin
This feature is mentioned in the spec attached to SPARK-18085 but there's not a
lot of discussion about it.
It would be good to implement incremental parsing of event logs in the SHS.
With the new work, UI data is stored on disk, so it should be possible to save
enough metadata about the event log and the state of the listeners to allow one
to resume parsing the log of a live application at the point where it stopped
in the previous iteration.
This would considerably speed up parsing on updates, and could be done
speculatively so that UIs for new applications are available in the SHS almost
immediately.
I'm filing this as a separate enhancement because I don't want to block
SPARK-18085 on this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
