[
https://issues.apache.org/jira/browse/SPARK-29579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-29579:
---------------------------------
Parent: (was: SPARK-28594)
Issue Type: Task (was: Sub-task)
> Guarantee compatibility of snapshot (live entities, KVstore entities)
> ---------------------------------------------------------------------
>
> Key: SPARK-29579
> URL: https://issues.apache.org/jira/browse/SPARK-29579
> Project: Spark
> Issue Type: Task
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> This issue is a follow-up issue after SPARK-29111 and SPARK-29261, which both
> issues WILL NOT guarantee compatibility.
> To safely clean up old event log files after snapshot has been written for
> these files, we have to ensure the snapshot file can restore the state as
> same as we replay from these event log files. The issue is on migrating to
> the newer Spark version - if snapshot is not readable due to incompatibility,
> the app cannot be read entirely as we've already removed old event log files.
> If we can guarantee compatibility we can move on to the next item, cleaning
> up old event log files to save space.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
