[
https://issues.apache.org/jira/browse/SPARK-29579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-29579.
----------------------------------
Resolution: Invalid
SPARK-29779 invalidates the needs of this. Closing.
> Guarantee compatibility of snapshot (live entities, KVstore entities)
> ---------------------------------------------------------------------
>
> Key: SPARK-29579
> URL: https://issues.apache.org/jira/browse/SPARK-29579
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> This issue is a follow-up issue after SPARK-29111 and SPARK-29261, which both
> issues WILL NOT guarantee compatibility.
> To safely clean up old event log files after snapshot has been written for
> these files, we have to ensure the snapshot file can restore the state as
> same as we replay from these event log files. The issue is on migrating to
> the newer Spark version - if snapshot is not readable due to incompatibility,
> the app cannot be read entirely as we've already removed old event log files.
> If we can guarantee compatibility we can move on to the next item, cleaning
> up old event log files to save space.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
