[
https://issues.apache.org/jira/browse/SPARK-33215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-33215.
----------------------------------
Fix Version/s: 3.1.0
Resolution: Fixed
Issue resolved by pull request 30126
[https://github.com/apache/spark/pull/30126]
> Speed up event log download by skipping UI rebuild
> --------------------------------------------------
>
> Key: SPARK-33215
> URL: https://issues.apache.org/jira/browse/SPARK-33215
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 2.4.7, 3.0.1
> Reporter: Baohe Zhang
> Assignee: Baohe Zhang
> Priority: Major
> Fix For: 3.1.0
>
>
> Right now, when we want to download the event logs from the spark history
> server(SHS), SHS will need to parse entire the event log to rebuild UI, and
> this is just for view permission checks. UI rebuilding is a time-consuming
> and memory-intensive task, especially for large logs. However, this process
> is unnecessary for event log download.
> This patch enables SHS to check UI view permissions of a given app/attempt
> for a given user, without rebuilding the UI. This is achieved by adding a
> method "checkUIViewPermissions(appId: String, attemptId: Option[String],
> user: String): Boolean" to many layers of history server components.
> With this patch, UI rebuild can be skipped when downloading event logs from
> the history server. Thus the time of downloading a GB scale event log can be
> reduced from several minutes to several seconds, and the memory consumption
> of UI rebuilding can be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
