I agree with you @Jingsong. Best, Yu Zelin
> 2023年5月30日 16:15，Jingsong Li <[email protected]> 写道： > > I think we can just throw exceptions for pure numeric tag names. > > Iceberg's behavior looks confusing. > > Best, > Jingsong > > On Tue, May 30, 2023 at 3:40 PM yu zelin <[email protected]> wrote: >> >> Hi, Shammon, >> >> An intuitive way is use numeric string to indicate snapshot and non-numeric >> string to indicate tag. >> For example: >> >> SELECT * FROM t VERSION AS OF 1 —to snapshot #1 >> SELECT * FROM t VERSION AS OF ‘last_year’ —to tag `last_year` >> >> This is also how iceberg do [1]. >> >> However, if we use this way, the tag name cannot be numeric string. I think >> this is acceptable and I will add this to the document. >> >> Best, >> Yu Zelin >> >> [1] https://iceberg.apache.org/docs/latest/spark-queries/#sql >> >>> 2023年5月30日 12:17，Shammon FY <[email protected]> 写道： >>> >>> Hi zelin, >>> >>> Thanks for your update. I have one comment about Time Travel on savepoint. >>> >>> Currently we can use statement in spark for specific snapshot 1 >>> SELECT * FROM t VERSION AS OF 1; >>> >>> My point is how can we distinguish between snapshot and savepoint when >>> users submit a statement as followed: >>> SELECT * FROM t VERSION AS OF <version value>; >>> >>> Best, >>> Shammon FY >>> >>> On Tue, May 30, 2023 at 11:37 AM yu zelin <[email protected]> wrote: >>> >>>> Hi, Jingsong, >>>> >>>> Thanks for your feedback. >>>> >>>> ## TAG ID >>>> It seems the id is useless currently. I’ll remove it. >>>> >>>> ## Time Travel Syntax >>>> Since tag id is removed, we can just use: >>>> >>>> SELECT * FROM t VERSION AS OF ’tag-name’ >>>> >>>> to travel to a tag. >>>> >>>> ## Tag class >>>> I agree with you that we can reuse the Snapshot class. We can introduce >>>> `TagManager` >>>> only to manage tags. >>>> >>>> ## Expiring Snapshot >>>>> why not record it in ManifestEntry? >>>> This is because every time Paimon generate a snapshot, it will create new >>>> ManifestEntries >>>> for data files. Consider this scenario, if we record it in ManifestEntry, >>>> assuming we commit >>>> data file A to snapshot #1, we will get manifest entry Entry#1 as [ADD, >>>> A, commit at #1]. >>>> Then we commit -A to snapshot #2, we will get manifest entry Entry#2 as >>>> [DELETE, A, ?], >>>> as you can see, we cannot know at which snapshot we commit the file A. So >>>> we have to >>>> record this information to data file meta directly. >>>> >>>>> We should note that "record it in `DataFileMeta` should be done before >>>> “tag” >>>> and document version compatibility. >>>> >>>> I will add message for this. >>>> >>>> Best, >>>> Yu Zelin >>>> >>>> >>>>> 2023年5月29日 10:29，Jingsong Li <[email protected]> 写道： >>>>> >>>>> Thanks Zelin for the update. >>>>> >>>>> ## TAG ID >>>>> >>>>> Is this useful? We have tag-name, snapshot-id, and now introducing a >>>>> tag id? What is used? >>>>> >>>>> ## Time Travel >>>>> >>>>> SELECT * FROM t VERSION AS OF tag-name.<name> >>>>> >>>>> This does not look like sql standard. >>>>> >>>>> Why do we introduce this `tag-name` prefix? >>>>> >>>>> ## Tag class >>>>> >>>>> Why not just use the Snapshot class? It looks like we don't need to >>>>> introduce Tag class. We can just copy the snapshot file to tag/. >>>>> >>>>> ## Expiring Snapshot >>>>> >>>>> We should note that "record it in `DataFileMeta`" should be done >>>>> before "tag". And document version compatibility. >>>>> And why not record it in ManifestEntry? >>>>> >>>>> Best, >>>>> Jingsong >>>>> >>>>> On Fri, May 26, 2023 at 11:15 AM yu zelin <[email protected]> wrote: >>>>>> >>>>>> Hi, all, >>>>>> >>>>>> FYI, I have updated the PIP [1]. >>>>>> >>>>>> Main changes: >>>>>> - Use new name `tag` >>>>>> - Enrich Motivation >>>>>> - New Section `Data Files Handling` to describe how to determine a data >>>> files can be deleted. >>>>>> >>>>>> Best, >>>>>> Yu Zelin >>>>>> >>>>>> [1] https://cwiki.apache.org/confluence/x/NxE0Dw >>>>>> >>>>>>> 2023年5月24日 17:18，yu zelin <[email protected]> 写道： >>>>>>> >>>>>>> Hi, Guojun, >>>>>>> >>>>>>> I’d like to share my thoughts about your questions. >>>>>>> >>>>>>> 1. Expiration of savepoint >>>>>>> In my opinion, savepoints are created in a long interval, so there >>>> will not exist too many of them. >>>>>>> If users create a savepoint per day, there are 365 savepoints a year. >>>> So I didn’t consider expiration >>>>>>> of it, and I think provide a flink action like `delete-savepoint id = >>>> 1` is enough now. >>>>>>> But if it is really important, we can introduce table options to do >>>> so. I think we can do it like expiring >>>>>>> snapshots. >>>>>>> >>>>>>> 2. > id of compacted snapshot picked by the savepoint >>>>>>> My initial idea is picking a compacted snapshot or doing compaction >>>> before creating savepoint. But >>>>>>> After discuss with Jingsong, I found it’s difficult. So now I suppose >>>> to directly create savepoint from >>>>>>> the given snapshot. Maybe we can optimize it later. >>>>>>> The changes will be updated soon. >>>>>>>> manifest file list in system-table >>>>>>> I think manifest file is not very important for users. Users can find >>>> when a savepoint is created, and >>>>>>> get the savepoint id, then they can query it from the savepoint by the >>>> id. I did’t see what scenario >>>>>>> the users need the manifest file information. What do you think? >>>>>>> >>>>>>> Best, >>>>>>> Yu Zelin >>>>>>> >>>>>>>> 2023年5月24日 10:50，Guojun Li <[email protected]> 写道： >>>>>>>> >>>>>>>> Thanks zelin for bringing up the discussion. I'm thinking about: >>>>>>>> 1. How to manage the savepoints if there are no expiration mechanism, >>>> by >>>>>>>> the TTL management of storages or external script? >>>>>>>> 2. I think the id of compacted snapshot picked by the savepoint and >>>>>>>> manifest file list is also important information for users, could >>>> these >>>>>>>> information be stored in the system-table? >>>>>>>> >>>>>>>> Best, >>>>>>>> Guojun >>>>>>>> >>>>>>>> On Mon, May 22, 2023 at 9:13 PM Jingsong Li <[email protected]> >>>> wrote: >>>>>>>> >>>>>>>>> FYI >>>>>>>>> >>>>>>>>> The PIP lacks a table to show Discussion thread & Vote thread & >>>> ISSUE... >>>>>>>>> >>>>>>>>> Best >>>>>>>>> Jingsong >>>>>>>>> >>>>>>>>> On Mon, May 22, 2023 at 4:48 PM yu zelin <[email protected]> >>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi, all, >>>>>>>>>> >>>>>>>>>> Thank all of you for your suggestions and questions. After reading >>>> your >>>>>>>>> suggestions, I adopt some of them and I want to share my opinions >>>> here. >>>>>>>>>> >>>>>>>>>> To make my statements more clear, I will still use the word >>>> `savepoint`. >>>>>>>>> When we make a consensus, the name may be changed. >>>>>>>>>> >>>>>>>>>> 1. The purposes of savepoint >>>>>>>>>> >>>>>>>>>> As Shammon mentioned, Flink and database also have the concept of >>>>>>>>> `savepoint`. So it’s better to clarify the purposes of our savepoint. >>>>>>>>> Thanks for Nicholas and Jingsong, I think your explanations are very >>>> clear. >>>>>>>>> I’d like to give my summary: >>>>>>>>>> >>>>>>>>>> (1) Fault recovery (or we can say disaster recovery). Users can ROLL >>>>>>>>> BACK to a savepoint if needed. If user rollbacks to a savepoint, the >>>> table >>>>>>>>> will hold the data in the savepoint and the data committed after the >>>>>>>>> savepoint will be deleted. In this scenario we need savepoint because >>>>>>>>> snapshots may have expired, the savepoint can keep longer and save >>>> user’s >>>>>>>>> old data. >>>>>>>>>> >>>>>>>>>> (2) Record versions of data at a longer interval (typically daily >>>> level >>>>>>>>> or weekly level). With savepoint, user can query the old data in >>>> batch >>>>>>>>> mode. Comparing to copy records to a new table or merge incremental >>>> records >>>>>>>>> with old records (like using merge into in Hive), the savepoint is >>>> more >>>>>>>>> lightweight because we don’t copy data files, we just record the >>>> meta data >>>>>>>>> of them. >>>>>>>>>> >>>>>>>>>> As you can see, savepoint is very similar to snapshot. The >>>> differences >>>>>>>>> are: >>>>>>>>>> >>>>>>>>>> (1) Savepoint lives longer. In most cases, snapshot’s life time is >>>>>>>>> about several minutes to hours. We suppose the savepoint can live >>>> several >>>>>>>>> days, weeks, or even months. >>>>>>>>>> >>>>>>>>>> (2) Savepoint is mainly used for batch reading for historical data. >>>> In >>>>>>>>> this PIP, we don’t introduce streaming reading for savepoint. >>>>>>>>>> >>>>>>>>>> 2. Candidates of name >>>>>>>>>> >>>>>>>>>> I agree with Jingsong that we can use a new name. Since the purpose >>>> and >>>>>>>>> mechanism (savepoint is very similar to snapshot) of savepoint is >>>> similar >>>>>>>>> to `tag` in iceberg, maybe we can use `tag`. >>>>>>>>>> >>>>>>>>>> In my opinion, an alternative is `anchor`. All the snapshots are >>>> like >>>>>>>>> the navigation path of the streaming data, and an `anchor` can stop >>>> it in a >>>>>>>>> place. >>>>>>>>>> >>>>>>>>>> 3. Public table operations and options >>>>>>>>>> >>>>>>>>>> We supposed to expose some operations and table options for user to >>>>>>>>> manage the savepoint. >>>>>>>>>> >>>>>>>>>> (1) Operations (Currently for Flink) >>>>>>>>>> We provide flink actions to manage savepoints: >>>>>>>>>> create-savepoint: To generate a savepoint from latest snapshot. >>>>>>>>> Support to create from specified snapshot. >>>>>>>>>> delete-savepoint: To delete specified savepoint. >>>>>>>>>> rollback-to: To roll back to a specified savepoint. >>>>>>>>>> >>>>>>>>>> (2) Table options >>>>>>>>>> We suppose to provide options for creating savepoint periodically: >>>>>>>>>> savepoint.create-time: When to create the savepoint. Example: 00:00 >>>>>>>>>> savepoint.create-interval: Interval between the creation of two >>>>>>>>> savepoints. Examples: 2 d. >>>>>>>>>> savepoint.time-retained: The maximum time of savepoints to retain. >>>>>>>>>> >>>>>>>>>> (3) Procedures (future work) >>>>>>>>>> Spark supports SQL extension. After we support Spark CALL >>>> statement, we >>>>>>>>> can provide procedures to create, delete or rollback to savepoint >>>> for Spark >>>>>>>>> users. >>>>>>>>>> >>>>>>>>>> Support of CALL is on the road map of Flink. In future version, we >>>> can >>>>>>>>> also support savepoint-related procedures for Flink users. >>>>>>>>>> >>>>>>>>>> 4. Expiration of data files >>>>>>>>>> >>>>>>>>>> Currently, when a snapshot is expired, data files that not be used >>>> by >>>>>>>>> other snapshots. After we introduce the savepoint, we must make sure >>>> the >>>>>>>>> data files saved by savepoint will not be deleted. >>>>>>>>>> >>>>>>>>>> Conversely, when a savepoint is deleted, the data files that not be >>>>>>>>> used by existing snapshots and other savepoints will be deleted. >>>>>>>>>> >>>>>>>>>> I have wrote some POC codes to implement it. I will update the >>>> mechanism >>>>>>>>> in PIP soon. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Yu Zelin >>>>>>>>>> >>>>>>>>>>> 2023年5月21日 20:54，Jingsong Li <[email protected]> 写道： >>>>>>>>>>> >>>>>>>>>>> Thanks Yun for your information. >>>>>>>>>>> >>>>>>>>>>> We need to be careful to avoid confusion between Paimon and Flink >>>>>>>>>>> concepts about "savepoint" >>>>>>>>>>> >>>>>>>>>>> Maybe we don't have to insist on using this "savepoint", for >>>> example, >>>>>>>>>>> TAG is also a candidate just like Iceberg [1] >>>>>>>>>>> >>>>>>>>>>> [1] https://iceberg.apache.org/docs/latest/branching/ >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Jingsong >>>>>>>>>>> >>>>>>>>>>> On Sun, May 21, 2023 at 8:51 PM Jingsong Li < >>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Thanks Nicholas for your detailed requirements. >>>>>>>>>>>> >>>>>>>>>>>> We need to supplement user requirements in FLIP, which is mainly >>>> aimed >>>>>>>>>>>> at two purposes: >>>>>>>>>>>> 1. Fault recovery for data errors (named: restore or rollback-to) >>>>>>>>>>>> 2. Used to record versions at the day level (such as), targeting >>>>>>>>> batch queries >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Jingsong >>>>>>>>>>>> >>>>>>>>>>>> On Sat, May 20, 2023 at 2:55 PM Yun Tang <[email protected]> >>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Guys, >>>>>>>>>>>>> >>>>>>>>>>>>> Since we use Paimon with Flink in most cases, I think we need to >>>>>>>>> identify the same word "savepoint" in different systems. >>>>>>>>>>>>> >>>>>>>>>>>>> For Flink, savepoint means: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Triggered by users, not periodically triggered by the system >>>>>>>>> itself. However, this FLIP wants to support it created periodically. >>>>>>>>>>>>> 2. Even the so-called incremental native savepoint [1], it will >>>>>>>>> not depend on the previous checkpoints or savepoints, it will still >>>> copy >>>>>>>>> files on DFS to the self-contained savepoint folder. However, from >>>> the >>>>>>>>> description of this FLIP about the deletion of expired snapshot >>>> files, >>>>>>>>> paimion savepoint will refer to the previously existing files >>>> directly. >>>>>>>>>>>>> >>>>>>>>>>>>> I don't think we need to make the semantics of Paimon totally the >>>>>>>>> same as Flink's. However, we need to introduce a table to tell the >>>>>>>>> difference compared with Flink and discuss about the difference. >>>>>>>>>>>>> >>>>>>>>>>>>> [1] >>>>>>>>> >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-203%3A+Incremental+savepoints#FLIP203:Incrementalsavepoints-Semantic >>>>>>>>>>>>> >>>>>>>>>>>>> Best >>>>>>>>>>>>> Yun Tang >>>>>>>>>>>>> ________________________________ >>>>>>>>>>>>> From: Nicholas Jiang <[email protected]> >>>>>>>>>>>>> Sent: Friday, May 19, 2023 17:40 >>>>>>>>>>>>> To: [email protected] <[email protected]> >>>>>>>>>>>>> Subject: Re: [DISCUSS] PIP-4 Support savepoint >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Guys, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks Zelin for driving the savepoint proposal. I propose some >>>>>>>>> opinions for savepont: >>>>>>>>>>>>> >>>>>>>>>>>>> -- About "introduce savepoint for Paimon to persist full data in >>>> a >>>>>>>>> time point" >>>>>>>>>>>>> >>>>>>>>>>>>> The motivation of savepoint proposal is more like snapshot TTL >>>>>>>>> management. Actually, disaster recovery is very much mission >>>> critical for >>>>>>>>> any software. Especially when it comes to data systems, the impact >>>> could be >>>>>>>>> very serious leading to delay in business decisions or even wrong >>>> business >>>>>>>>> decisions at times. Savepoint is proposed to assist users in >>>> recovering >>>>>>>>> data from a previous state: "savepoint" and "restore". >>>>>>>>>>>>> >>>>>>>>>>>>> "savepoint" saves the Paimon table as of the commit time, >>>> therefore >>>>>>>>> if there is a savepoint, the data generated in the corresponding >>>> commit >>>>>>>>> could not be clean. Meanwhile, savepoint could let user restore the >>>> table >>>>>>>>> to this savepoint at a later point in time if need be. On similar >>>> lines, >>>>>>>>> savepoint cannot be triggered on a commit that is already cleaned up. >>>>>>>>> Savepoint is synonymous to taking a backup, just that we don't make >>>> a new >>>>>>>>> copy of the table, but just save the state of the table elegantly so >>>> that >>>>>>>>> we can restore it later when in need. >>>>>>>>>>>>> >>>>>>>>>>>>> "restore" lets you restore your table to one of the savepoint >>>>>>>>> commit. Meanwhile, it cannot be undone (or reversed) and so care >>>> should be >>>>>>>>> taken before doing a restore. At this time, Paimon would delete all >>>> data >>>>>>>>> files and commit files (timeline files) greater than the savepoint >>>> commit >>>>>>>>> to which the table is being restored. >>>>>>>>>>>>> >>>>>>>>>>>>> BTW, it's better to introduce snapshot view based on savepoint, >>>>>>>>> which could improve query performance of historical data for Paimon >>>> table. >>>>>>>>>>>>> >>>>>>>>>>>>> -- About Public API of savepont >>>>>>>>>>>>> >>>>>>>>>>>>> Current introduced savepoint interfaces in Public API are not >>>> enough >>>>>>>>> for users, for example, deleteSavepoint, restoreSavepoint etc. >>>>>>>>>>>>> >>>>>>>>>>>>> -- About "Paimon's savepoint need to be combined with Flink's >>>>>>>>> savepoint": >>>>>>>>>>>>> >>>>>>>>>>>>> If paimon supports savepoint mechanism and provides savepoint >>>>>>>>> interfaces, the integration with Flink's savepoint is not blocked >>>> for this >>>>>>>>> proposal. >>>>>>>>>>>>> >>>>>>>>>>>>> In summary, savepoint is not only used to improve the query >>>>>>>>> performance of historical data, but also used for disaster recovery >>>>>>>>> processing. >>>>>>>>>>>>> >>>>>>>>>>>>> On 2023/05/17 09:53:11 Jingsong Li wrote: >>>>>>>>>>>>>> What Shammon mentioned is interesting. I agree with what he said >>>>>>>>> about >>>>>>>>>>>>>> the differences in savepoints between databases and stream >>>>>>>>> computing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> About "Paimon's savepoint need to be combined with Flink's >>>>>>>>> savepoint": >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think it is possible, but we may need to deal with this in >>>> another >>>>>>>>>>>>>> mechanism, because the snapshots after savepoint may expire. We >>>> need >>>>>>>>>>>>>> to compare data between two savepoints to generate incremental >>>> data >>>>>>>>>>>>>> for streaming read. >>>>>>>>>>>>>> >>>>>>>>>>>>>> But this may not need to block FLIP, it looks like the current >>>>>>>>> design >>>>>>>>>>>>>> does not break the future combination? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> Jingsong >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, May 17, 2023 at 5:33 PM Shammon FY <[email protected]> >>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Caizhi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for your comments. As you mentioned, I think we may >>>> need to >>>>>>>>> discuss >>>>>>>>>>>>>>> the role of savepoint in Paimon. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If I understand correctly, the main feature of savepoint in the >>>>>>>>> current PIP >>>>>>>>>>>>>>> is that the savepoint will not be expired, and users can >>>> perform a >>>>>>>>> query on >>>>>>>>>>>>>>> the savepoint according to time-travel. Besides that, there is >>>>>>>>> savepoint in >>>>>>>>>>>>>>> the database and Flink. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Savepoint in database. The database can roll back table >>>> data to >>>>>>>>> the >>>>>>>>>>>>>>> specified 'version' based on savepoint. So the key point of >>>>>>>>> savepoint in >>>>>>>>>>>>>>> the database is to rollback data. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. Savepoint in Flink. Users can trigger a savepoint with a >>>>>>>>> specific >>>>>>>>>>>>>>> 'path', and save all data of state to the savepoint for job. >>>> Then >>>>>>>>> users can >>>>>>>>>>>>>>> create a new job based on the savepoint to continue consuming >>>>>>>>> incremental >>>>>>>>>>>>>>> data. I think the core capabilities are: backup for a job, and >>>>>>>>> resume a job >>>>>>>>>>>>>>> based on the savepoint. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In addition to the above, Paimon may also face data write >>>>>>>>> corruption and >>>>>>>>>>>>>>> need to recover data based on the specified savepoint. So we >>>> may >>>>>>>>> need to >>>>>>>>>>>>>>> consider what abilities should Paimon savepoint need besides >>>> the >>>>>>>>> ones >>>>>>>>>>>>>>> mentioned in the current PIP? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Additionally, as mentioned above, Flink also has >>>>>>>>>>>>>>> savepoint mechanism. During the process of streaming data from >>>>>>>>> Flink to >>>>>>>>>>>>>>> Paimon, does Paimon's savepoint need to be combined with >>>> Flink's >>>>>>>>> savepoint? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> Shammon FY >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, May 17, 2023 at 4:02 PM Caizhi Weng < >>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi developers! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks Zelin for bringing up the discussion. The proposal >>>> seems >>>>>>>>> good to me >>>>>>>>>>>>>>>> overall. However I'd also like to bring up a few options. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. As Jingsong mentioned, Savepoint class should not become a >>>>>>>>> public API, >>>>>>>>>>>>>>>> at least for now. What we need to discuss for the public API >>>> is >>>>>>>>> how the >>>>>>>>>>>>>>>> users can create or delete savepoints. For example, what the >>>>>>>>> table option >>>>>>>>>>>>>>>> looks like, what commands and options are provided for the >>>> Flink >>>>>>>>> action, >>>>>>>>>>>>>>>> etc. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. Currently most Flink actions are related to streaming >>>>>>>>> processing, so >>>>>>>>>>>>>>>> only Flink can support them. However, savepoint creation and >>>>>>>>> deletion seems >>>>>>>>>>>>>>>> like a feature for batch processing. So aside from Flink >>>> actions, >>>>>>>>> shall we >>>>>>>>>>>>>>>> also provide something like Spark actions for savepoints? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would also like to comment on Shammon's views. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Should we introduce an option for savepoint path which may be >>>>>>>>> different >>>>>>>>>>>>>>>>> from 'warehouse'? Then users can backup the data of >>>> savepoint. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I don't see this is necessary. To backup a table the user just >>>>>>>>> need to copy >>>>>>>>>>>>>>>> all files from the table directory. Savepoint in Paimon, as >>>> far >>>>>>>>> as I >>>>>>>>>>>>>>>> understand, is mainly for users to review historical data, not >>>>>>>>> for backing >>>>>>>>>>>>>>>> up tables. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Will the savepoint copy data files from snapshot or only save >>>>>>>>> meta files? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It would be a heavy burden if a savepoint copies all its >>>> files. >>>>>>>>> As I >>>>>>>>>>>>>>>> mentioned above, savepoint is not for backing up tables. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> How can users create a new table and restore data from the >>>>>>>>> specified >>>>>>>>>>>>>>>>> savepoint? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This reminds me of savepoints in Flink. Still, savepoint is >>>> not >>>>>>>>> for backing >>>>>>>>>>>>>>>> up tables so I guess we don't need to support "restoring data" >>>>>>>>> from a >>>>>>>>>>>>>>>> savepoint. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Shammon FY <[email protected]> 于2023年5月17日周三 10:32写道： >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks Zelin for initiating this discussion. I have some >>>>>>>>> comments: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. Should we introduce an option for savepoint path which >>>> may be >>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>> from 'warehouse'? Then users can backup the data of >>>> savepoint. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. Will the savepoint copy data files from snapshot or only >>>> save >>>>>>>>> meta >>>>>>>>>>>>>>>>> files? The description in the PIP "After we introduce >>>> savepoint, >>>>>>>>> we >>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>> also check if the data files are used by savepoints." looks >>>> like >>>>>>>>> we only >>>>>>>>>>>>>>>>> save meta files for savepoint. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3. How can users create a new table and restore data from the >>>>>>>>> specified >>>>>>>>>>>>>>>>> savepoint? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> Shammon FY >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, May 17, 2023 at 10:19 AM Jingsong Li < >>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks Zelin for driving. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Some comments: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. I think it's possible to advance `Proposed Changes` to >>>> the >>>>>>>>> top, >>>>>>>>>>>>>>>>>> Public API has no meaning if I don't know how to do it. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2. Public API, Savepoint and SavepointManager are not Public >>>>>>>>> API, only >>>>>>>>>>>>>>>>>> Flink action or configuration option should be public API. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 3.Maybe we can have a separate chapter to describe >>>>>>>>>>>>>>>>>> `savepoint.create-interval`, maybe 'Periodically >>>> savepoint'? It >>>>>>>>> is not >>>>>>>>>>>>>>>>>> just an interval, because the true user case is savepoint >>>> after >>>>>>>>> 0:00. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 4.About 'Interaction with Snapshot', to be continued ... >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>> Jingsong >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, May 16, 2023 at 7:07 PM yu zelin < >>>> [email protected] >>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi, Paimon Devs, >>>>>>>>>>>>>>>>>>> I’d like to start a discussion about PIP-4[1]. In this >>>>>>>>> PIP, I >>>>>>>>>>>>>>>> want >>>>>>>>>>>>>>>>>> to talk about why we need savepoint, and some thoughts about >>>>>>>>> managing >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> using savepoint. Look forward to your question and >>>> suggestions. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>> Yu Zelin >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/confluence/x/NxE0Dw >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>
