[
https://issues.apache.org/jira/browse/SPARK-60756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18135359#comment-18135359
]
Yuqin Luo commented on SPARK-60756:
-----------------------------------
PR: https://github.com/apache/spark/pull/59964
> SQLAppStatusListener.onJobStart forces synchronous kvstore write during
> history replay, causing O(J^2) replay cost
> ------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60756
> URL: https://issues.apache.org/jira/browse/SPARK-60756
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.6
> Reporter: Yuqin Luo
> Priority: Major
> Labels: pull-request-available
>
> SQLAppStatusListener.onJobStart unconditionally calls update(exec, force =
> true), forcing a synchronous KVStore write on every job-start event, even
> during History Server replay (live = false).
> During replay, liveUpdatePeriodNs is -1, so this forced write is the only
> source of intermediate writes. Each write serializes the execution's growing
> job/stage state, so an execution with J job-start events costs O(J^2) to
> replay.
> Proposed fix: change update(exec, force = true) to update(exec, force = live)
> in onJobStart. Live UI behavior is unchanged. During replay, the final write
> on execution end (onExecutionEnd) is preserved, and executions that never
> complete are still persisted by the existing kvstore.onFlush hook.
> Measured with local History Server replay of two real production event logs
> (patched vs. unpatched):
> - 615.7 MB event log: 531s -> 101s (5.3x faster)
> - 1010 MB event log: 833.7s -> 284.5s (2.93x faster)
> The final replayed data is unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
