FYI The PIP lacks a table to show Discussion thread & Vote thread & ISSUE...
Best Jingsong On Mon, May 22, 2023 at 4:48 PM yu zelin <[email protected]> wrote: > > Hi, all, > > Thank all of you for your suggestions and questions. After reading your > suggestions, I adopt some of them and I want to share my opinions here. > > To make my statements more clear, I will still use the word `savepoint`. When > we make a consensus, the name may be changed. > > 1. The purposes of savepoint > > As Shammon mentioned, Flink and database also have the concept of > `savepoint`. So it’s better to clarify the purposes of our savepoint. Thanks > for Nicholas and Jingsong, I think your explanations are very clear. I’d like > to give my summary: > > (1) Fault recovery (or we can say disaster recovery). Users can ROLL BACK to > a savepoint if needed. If user rollbacks to a savepoint, the table will hold > the data in the savepoint and the data committed after the savepoint will be > deleted. In this scenario we need savepoint because snapshots may have > expired, the savepoint can keep longer and save user’s old data. > > (2) Record versions of data at a longer interval (typically daily level or > weekly level). With savepoint, user can query the old data in batch mode. > Comparing to copy records to a new table or merge incremental records with > old records (like using merge into in Hive), the savepoint is more > lightweight because we don’t copy data files, we just record the meta data of > them. > > As you can see, savepoint is very similar to snapshot. The differences are: > > (1) Savepoint lives longer. In most cases, snapshot’s life time is about > several minutes to hours. We suppose the savepoint can live several days, > weeks, or even months. > > (2) Savepoint is mainly used for batch reading for historical data. In this > PIP, we don’t introduce streaming reading for savepoint. > > 2. Candidates of name > > I agree with Jingsong that we can use a new name. Since the purpose and > mechanism (savepoint is very similar to snapshot) of savepoint is similar to > `tag` in iceberg, maybe we can use `tag`. > > In my opinion, an alternative is `anchor`. All the snapshots are like the > navigation path of the streaming data, and an `anchor` can stop it in a place. > > 3. Public table operations and options > > We supposed to expose some operations and table options for user to manage > the savepoint. > > (1) Operations (Currently for Flink) > We provide flink actions to manage savepoints: > create-savepoint: To generate a savepoint from latest snapshot. Support > to create from specified snapshot. > delete-savepoint: To delete specified savepoint. > rollback-to: To roll back to a specified savepoint. > > (2) Table options > We suppose to provide options for creating savepoint periodically: > savepoint.create-time: When to create the savepoint. Example: 00:00 > savepoint.create-interval: Interval between the creation of two > savepoints. Examples: 2 d. > savepoint.time-retained: The maximum time of savepoints to retain. > > (3) Procedures (future work) > Spark supports SQL extension. After we support Spark CALL statement, we can > provide procedures to create, delete or rollback to savepoint for Spark users. > > Support of CALL is on the road map of Flink. In future version, we can also > support savepoint-related procedures for Flink users. > > 4. Expiration of data files > > Currently, when a snapshot is expired, data files that not be used by other > snapshots. After we introduce the savepoint, we must make sure the data files > saved by savepoint will not be deleted. > > Conversely, when a savepoint is deleted, the data files that not be used by > existing snapshots and other savepoints will be deleted. > > I have wrote some POC codes to implement it. I will update the mechanism in > PIP soon. > > Best, > Yu Zelin > > > 2023年5月21日 20:54，Jingsong Li <[email protected]> 写道： > > > > Thanks Yun for your information. > > > > We need to be careful to avoid confusion between Paimon and Flink > > concepts about "savepoint" > > > > Maybe we don't have to insist on using this "savepoint", for example, > > TAG is also a candidate just like Iceberg [1] > > > > [1] https://iceberg.apache.org/docs/latest/branching/ > > > > Best, > > Jingsong > > > > On Sun, May 21, 2023 at 8:51 PM Jingsong Li <[email protected]> wrote: > >> > >> Thanks Nicholas for your detailed requirements. > >> > >> We need to supplement user requirements in FLIP, which is mainly aimed > >> at two purposes: > >> 1. Fault recovery for data errors (named: restore or rollback-to) > >> 2. Used to record versions at the day level (such as), targeting batch > >> queries > >> > >> Best, > >> Jingsong > >> > >> On Sat, May 20, 2023 at 2:55 PM Yun Tang <[email protected]> wrote: > >>> > >>> Hi Guys, > >>> > >>> Since we use Paimon with Flink in most cases, I think we need to identify > >>> the same word "savepoint" in different systems. > >>> > >>> For Flink, savepoint means: > >>> > >>> 1. Triggered by users, not periodically triggered by the system itself. > >>> However, this FLIP wants to support it created periodically. > >>> 2. Even the so-called incremental native savepoint [1], it will not > >>> depend on the previous checkpoints or savepoints, it will still copy > >>> files on DFS to the self-contained savepoint folder. However, from the > >>> description of this FLIP about the deletion of expired snapshot files, > >>> paimion savepoint will refer to the previously existing files directly. > >>> > >>> I don't think we need to make the semantics of Paimon totally the same as > >>> Flink's. However, we need to introduce a table to tell the difference > >>> compared with Flink and discuss about the difference. > >>> > >>> [1] > >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-203%3A+Incremental+savepoints#FLIP203:Incrementalsavepoints-Semantic > >>> > >>> Best > >>> Yun Tang > >>> ________________________________ > >>> From: Nicholas Jiang <[email protected]> > >>> Sent: Friday, May 19, 2023 17:40 > >>> To: [email protected] <[email protected]> > >>> Subject: Re: [DISCUSS] PIP-4 Support savepoint > >>> > >>> Hi Guys, > >>> > >>> Thanks Zelin for driving the savepoint proposal. I propose some opinions > >>> for savepont: > >>> > >>> -- About "introduce savepoint for Paimon to persist full data in a time > >>> point" > >>> > >>> The motivation of savepoint proposal is more like snapshot TTL > >>> management. Actually, disaster recovery is very much mission critical for > >>> any software. Especially when it comes to data systems, the impact could > >>> be very serious leading to delay in business decisions or even wrong > >>> business decisions at times. Savepoint is proposed to assist users in > >>> recovering data from a previous state: "savepoint" and "restore". > >>> > >>> "savepoint" saves the Paimon table as of the commit time, therefore if > >>> there is a savepoint, the data generated in the corresponding commit > >>> could not be clean. Meanwhile, savepoint could let user restore the table > >>> to this savepoint at a later point in time if need be. On similar lines, > >>> savepoint cannot be triggered on a commit that is already cleaned up. > >>> Savepoint is synonymous to taking a backup, just that we don't make a new > >>> copy of the table, but just save the state of the table elegantly so that > >>> we can restore it later when in need. > >>> > >>> "restore" lets you restore your table to one of the savepoint commit. > >>> Meanwhile, it cannot be undone (or reversed) and so care should be taken > >>> before doing a restore. At this time, Paimon would delete all data files > >>> and commit files (timeline files) greater than the savepoint commit to > >>> which the table is being restored. > >>> > >>> BTW, it's better to introduce snapshot view based on savepoint, which > >>> could improve query performance of historical data for Paimon table. > >>> > >>> -- About Public API of savepont > >>> > >>> Current introduced savepoint interfaces in Public API are not enough for > >>> users, for example, deleteSavepoint, restoreSavepoint etc. > >>> > >>> -- About "Paimon's savepoint need to be combined with Flink's savepoint": > >>> > >>> If paimon supports savepoint mechanism and provides savepoint interfaces, > >>> the integration with Flink's savepoint is not blocked for this proposal. > >>> > >>> In summary, savepoint is not only used to improve the query performance > >>> of historical data, but also used for disaster recovery processing. > >>> > >>> On 2023/05/17 09:53:11 Jingsong Li wrote: > >>>> What Shammon mentioned is interesting. I agree with what he said about > >>>> the differences in savepoints between databases and stream computing. > >>>> > >>>> About "Paimon's savepoint need to be combined with Flink's savepoint": > >>>> > >>>> I think it is possible, but we may need to deal with this in another > >>>> mechanism, because the snapshots after savepoint may expire. We need > >>>> to compare data between two savepoints to generate incremental data > >>>> for streaming read. > >>>> > >>>> But this may not need to block FLIP, it looks like the current design > >>>> does not break the future combination? > >>>> > >>>> Best, > >>>> Jingsong > >>>> > >>>> On Wed, May 17, 2023 at 5:33 PM Shammon FY <[email protected]> wrote: > >>>>> > >>>>> Hi Caizhi, > >>>>> > >>>>> Thanks for your comments. As you mentioned, I think we may need to > >>>>> discuss > >>>>> the role of savepoint in Paimon. > >>>>> > >>>>> If I understand correctly, the main feature of savepoint in the current > >>>>> PIP > >>>>> is that the savepoint will not be expired, and users can perform a > >>>>> query on > >>>>> the savepoint according to time-travel. Besides that, there is > >>>>> savepoint in > >>>>> the database and Flink. > >>>>> > >>>>> 1. Savepoint in database. The database can roll back table data to the > >>>>> specified 'version' based on savepoint. So the key point of savepoint in > >>>>> the database is to rollback data. > >>>>> > >>>>> 2. Savepoint in Flink. Users can trigger a savepoint with a specific > >>>>> 'path', and save all data of state to the savepoint for job. Then users > >>>>> can > >>>>> create a new job based on the savepoint to continue consuming > >>>>> incremental > >>>>> data. I think the core capabilities are: backup for a job, and resume a > >>>>> job > >>>>> based on the savepoint. > >>>>> > >>>>> In addition to the above, Paimon may also face data write corruption and > >>>>> need to recover data based on the specified savepoint. So we may need to > >>>>> consider what abilities should Paimon savepoint need besides the ones > >>>>> mentioned in the current PIP? > >>>>> > >>>>> Additionally, as mentioned above, Flink also has > >>>>> savepoint mechanism. During the process of streaming data from Flink to > >>>>> Paimon, does Paimon's savepoint need to be combined with Flink's > >>>>> savepoint? > >>>>> > >>>>> > >>>>> Best, > >>>>> Shammon FY > >>>>> > >>>>> > >>>>> On Wed, May 17, 2023 at 4:02 PM Caizhi Weng <[email protected]> > >>>>> wrote: > >>>>> > >>>>>> Hi developers! > >>>>>> > >>>>>> Thanks Zelin for bringing up the discussion. The proposal seems good > >>>>>> to me > >>>>>> overall. However I'd also like to bring up a few options. > >>>>>> > >>>>>> 1. As Jingsong mentioned, Savepoint class should not become a public > >>>>>> API, > >>>>>> at least for now. What we need to discuss for the public API is how the > >>>>>> users can create or delete savepoints. For example, what the table > >>>>>> option > >>>>>> looks like, what commands and options are provided for the Flink > >>>>>> action, > >>>>>> etc. > >>>>>> > >>>>>> 2. Currently most Flink actions are related to streaming processing, so > >>>>>> only Flink can support them. However, savepoint creation and deletion > >>>>>> seems > >>>>>> like a feature for batch processing. So aside from Flink actions, > >>>>>> shall we > >>>>>> also provide something like Spark actions for savepoints? > >>>>>> > >>>>>> I would also like to comment on Shammon's views. > >>>>>> > >>>>>> Should we introduce an option for savepoint path which may be different > >>>>>>> from 'warehouse'? Then users can backup the data of savepoint. > >>>>>>> > >>>>>> > >>>>>> I don't see this is necessary. To backup a table the user just need to > >>>>>> copy > >>>>>> all files from the table directory. Savepoint in Paimon, as far as I > >>>>>> understand, is mainly for users to review historical data, not for > >>>>>> backing > >>>>>> up tables. > >>>>>> > >>>>>> Will the savepoint copy data files from snapshot or only save meta > >>>>>> files? > >>>>>>> > >>>>>> > >>>>>> It would be a heavy burden if a savepoint copies all its files. As I > >>>>>> mentioned above, savepoint is not for backing up tables. > >>>>>> > >>>>>> How can users create a new table and restore data from the specified > >>>>>>> savepoint? > >>>>>> > >>>>>> > >>>>>> This reminds me of savepoints in Flink. Still, savepoint is not for > >>>>>> backing > >>>>>> up tables so I guess we don't need to support "restoring data" from a > >>>>>> savepoint. > >>>>>> > >>>>>> Shammon FY <[email protected]> 于2023年5月17日周三 10:32写道： > >>>>>> > >>>>>>> Thanks Zelin for initiating this discussion. I have some comments: > >>>>>>> > >>>>>>> 1. Should we introduce an option for savepoint path which may be > >>>>>> different > >>>>>>> from 'warehouse'? Then users can backup the data of savepoint. > >>>>>>> > >>>>>>> 2. Will the savepoint copy data files from snapshot or only save meta > >>>>>>> files? The description in the PIP "After we introduce savepoint, we > >>>>>> should > >>>>>>> also check if the data files are used by savepoints." looks like we > >>>>>>> only > >>>>>>> save meta files for savepoint. > >>>>>>> > >>>>>>> 3. How can users create a new table and restore data from the > >>>>>>> specified > >>>>>>> savepoint? > >>>>>>> > >>>>>>> Best, > >>>>>>> Shammon FY > >>>>>>> > >>>>>>> > >>>>>>> On Wed, May 17, 2023 at 10:19 AM Jingsong Li <[email protected]> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Thanks Zelin for driving. > >>>>>>>> > >>>>>>>> Some comments: > >>>>>>>> > >>>>>>>> 1. I think it's possible to advance `Proposed Changes` to the top, > >>>>>>>> Public API has no meaning if I don't know how to do it. > >>>>>>>> > >>>>>>>> 2. Public API, Savepoint and SavepointManager are not Public API, > >>>>>>>> only > >>>>>>>> Flink action or configuration option should be public API. > >>>>>>>> > >>>>>>>> 3.Maybe we can have a separate chapter to describe > >>>>>>>> `savepoint.create-interval`, maybe 'Periodically savepoint'? It is > >>>>>>>> not > >>>>>>>> just an interval, because the true user case is savepoint after 0:00. > >>>>>>>> > >>>>>>>> 4.About 'Interaction with Snapshot', to be continued ... > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Jingsong > >>>>>>>> > >>>>>>>> On Tue, May 16, 2023 at 7:07 PM yu zelin <[email protected]> > >>>>>> wrote: > >>>>>>>>> > >>>>>>>>> Hi, Paimon Devs, > >>>>>>>>> I’d like to start a discussion about PIP-4[1]. In this PIP, I > >>>>>> want > >>>>>>>> to talk about why we need savepoint, and some thoughts about managing > >>>>>> and > >>>>>>>> using savepoint. Look forward to your question and suggestions. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Yu Zelin > >>>>>>>>> > >>>>>>>>> [1] https://cwiki.apache.org/confluence/x/NxE0Dw > >>>>>>>> > >>>>>>> > >>>>>> > >>>> >
