[jira] [Updated] (SPARK-28870) Snapshot event log files to support incremental reading
[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28870: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Snapshot event log files to support incremental reading > --- > > Key: SPARK-28870 > URL: https://issues.apache.org/jira/browse/SPARK-28870 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on snapshotting the current status of > KVStore/AppStatusListener and enable incremental reading to speed up > replaying event logs. > This issue will be on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will > add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add > the ability to snapshot/restore of state of (SQL)AppStatusListeners. > Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any > compatibility, and this issue should deal with the situation: snapshot > version is not compatible with SHS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28870) Snapshot event log files to support incremental reading
[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-28870: - Description: This issue tracks the effort on snapshotting the current status of KVStore/AppStatusListener and enable incremental reading to speed up replaying event logs. This issue will be on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add the ability to snapshot/restore of state of (SQL)AppStatusListeners. Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any compatibility, and this issue should deal with the situation: snapshot version is not compatible with SHS. was: This issue tracks the effort on snapshotting the current status of KVStore/AppStatusListener and enable incremental reading to speed up replaying event logs. This issue will be on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add the ability to snapshot/restore of state of (SQL)AppStatusListeners. Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any compatibility, and this issue should deal with two situations as well: 1) some part of event log files got deleted 2) snapshot version is not compatible with SHS. > Snapshot event log files to support incremental reading > --- > > Key: SPARK-28870 > URL: https://issues.apache.org/jira/browse/SPARK-28870 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on snapshotting the current status of > KVStore/AppStatusListener and enable incremental reading to speed up > replaying event logs. > This issue will be on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will > add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add > the ability to snapshot/restore of state of (SQL)AppStatusListeners. > Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any > compatibility, and this issue should deal with the situation: snapshot > version is not compatible with SHS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28870) Snapshot event log files to support incremental reading
[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-28870: - Description: This issue tracks the effort on snapshotting the current status of KVStore/AppStatusListener and enable incremental reading to speed up replaying event logs. This issue will be on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add the ability to snapshot/restore of state of (SQL)AppStatusListeners. Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any compatibility, and this issue should deal with two situations as well: 1) some part of event log files got deleted 2) snapshot version is not compatible with SHS. was: This issue tracks the effort on compacting event log files into snapshot and enable incremental reading to speed up replaying event logs. This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled event log files. This issue will be also on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add the ability to snapshot/restore of state of (SQL)AppStatusListeners. Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any compatibility, and this issue should deal with two situations as well: 1) some part of event log files got deleted 2) snapshot version is not compatible with SHS. > Snapshot event log files to support incremental reading > --- > > Key: SPARK-28870 > URL: https://issues.apache.org/jira/browse/SPARK-28870 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on snapshotting the current status of > KVStore/AppStatusListener and enable incremental reading to speed up > replaying event logs. > This issue will be on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will > add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add > the ability to snapshot/restore of state of (SQL)AppStatusListeners. > Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any > compatibility, and this issue should deal with two situations as well: 1) > some part of event log files got deleted 2) snapshot version is not > compatible with SHS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28870) Snapshot event log files to support incremental reading
[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-28870: - Parent: (was: SPARK-28594) Issue Type: New Feature (was: Sub-task) > Snapshot event log files to support incremental reading > --- > > Key: SPARK-28870 > URL: https://issues.apache.org/jira/browse/SPARK-28870 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on compacting event log files into snapshot and > enable incremental reading to speed up replaying event logs. > This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled > event log files. This issue will be also on top of SPARK-29111 and > SPARK-29261, as SPARK-29111 will add the ability to snapshot/restore from/to > KVStore and SPARK-29261 will add the ability to snapshot/restore of state of > (SQL)AppStatusListeners. > Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any > compatibility, and this issue should deal with two situations as well: 1) > some part of event log files got deleted 2) snapshot version is not > compatible with SHS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28870) Snapshot event log files to support incremental reading
[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-28870: - Description: This issue tracks the effort on compacting event log files into snapshot and enable incremental reading to speed up replaying event logs. This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled event log files. This issue will be also on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add the ability to snapshot/restore of state of (SQL)AppStatusListeners. was: This issue tracks the effort on compacting event log files into snapshot and enable incremental reading to speed up replaying event logs. This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled event log files. This issue will be also on top of SPARK-29111 as SPARK-29111 will add the ability to snapshot/restore from/to KVStore. > Snapshot event log files to support incremental reading > --- > > Key: SPARK-28870 > URL: https://issues.apache.org/jira/browse/SPARK-28870 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on compacting event log files into snapshot and > enable incremental reading to speed up replaying event logs. > This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled > event log files. This issue will be also on top of SPARK-29111 and > SPARK-29261, as SPARK-29111 will add the ability to snapshot/restore from/to > KVStore and SPARK-29261 will add the ability to snapshot/restore of state of > (SQL)AppStatusListeners. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28870) Snapshot event log files to support incremental reading
[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-28870: - Description: This issue tracks the effort on compacting event log files into snapshot and enable incremental reading to speed up replaying event logs. This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled event log files. This issue will be also on top of SPARK-29111 as SPARK-29111 will add the ability to snapshot/restore from/to KVStore. was: This issue tracks the effort on compacting old event log files into snapshot and achieve both goals, 1) reduce overall event log file size 2) speed up replaying event logs. It also deals with cleaning event log files as snapshot will provide the safe way to clean up old event log files without losing ability to replay whole event logs. This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled event log files. This issue will be also on top of SPARK-29111 as SPARK-29111 will add the ability to snapshot/restore from/to KVStore. > Snapshot event log files to support incremental reading > --- > > Key: SPARK-28870 > URL: https://issues.apache.org/jira/browse/SPARK-28870 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on compacting event log files into snapshot and > enable incremental reading to speed up replaying event logs. > This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled > event log files. This issue will be also on top of SPARK-29111 as SPARK-29111 > will add the ability to snapshot/restore from/to KVStore. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28870) Snapshot event log files to support incremental reading
[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-28870: - Summary: Snapshot event log files to support incremental reading (was: Snapshot old event log files to support compaction) > Snapshot event log files to support incremental reading > --- > > Key: SPARK-28870 > URL: https://issues.apache.org/jira/browse/SPARK-28870 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on compacting old event log files into snapshot > and achieve both goals, 1) reduce overall event log file size 2) speed up > replaying event logs. It also deals with cleaning event log files as snapshot > will provide the safe way to clean up old event log files without losing > ability to replay whole event logs. > This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled > event log files. This issue will be also on top of SPARK-29111 as SPARK-29111 > will add the ability to snapshot/restore from/to KVStore. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
