+1 to vote
On Tue, May 30, 2023 at 6:22 PM yu zelin <[email protected]> wrote: > > Hi, all, > > Does anyone have questions or feedbacks? > > I will wait a while for your reply. If no, I’d like to start a vote later. > > Best, > Yu Zelin > > > 2023年5月30日 16:19，yu zelin <[email protected]> 写道： > > > > I agree with you @Jingsong. > > > > Best, > > Yu Zelin > > > >> 2023年5月30日 16:15，Jingsong Li <[email protected]> 写道： > >> > >> I think we can just throw exceptions for pure numeric tag names. > >> > >> Iceberg's behavior looks confusing. > >> > >> Best, > >> Jingsong > >> > >> On Tue, May 30, 2023 at 3:40 PM yu zelin <[email protected]> wrote: > >>> > >>> Hi, Shammon, > >>> > >>> An intuitive way is use numeric string to indicate snapshot and > >>> non-numeric string to indicate tag. > >>> For example: > >>> > >>> SELECT * FROM t VERSION AS OF 1 —to snapshot #1 > >>> SELECT * FROM t VERSION AS OF ‘last_year’ —to tag `last_year` > >>> > >>> This is also how iceberg do [1]. > >>> > >>> However, if we use this way, the tag name cannot be numeric string. I > >>> think this is acceptable and I will add this to the document. > >>> > >>> Best, > >>> Yu Zelin > >>> > >>> [1] https://iceberg.apache.org/docs/latest/spark-queries/#sql > >>> > >>>> 2023年5月30日 12:17，Shammon FY <[email protected]> 写道： > >>>> > >>>> Hi zelin, > >>>> > >>>> Thanks for your update. I have one comment about Time Travel on > >>>> savepoint. > >>>> > >>>> Currently we can use statement in spark for specific snapshot 1 > >>>> SELECT * FROM t VERSION AS OF 1; > >>>> > >>>> My point is how can we distinguish between snapshot and savepoint when > >>>> users submit a statement as followed: > >>>> SELECT * FROM t VERSION AS OF <version value>; > >>>> > >>>> Best, > >>>> Shammon FY > >>>> > >>>> On Tue, May 30, 2023 at 11:37 AM yu zelin <[email protected]> wrote: > >>>> > >>>>> Hi, Jingsong, > >>>>> > >>>>> Thanks for your feedback. > >>>>> > >>>>> ## TAG ID > >>>>> It seems the id is useless currently. I’ll remove it. > >>>>> > >>>>> ## Time Travel Syntax > >>>>> Since tag id is removed, we can just use: > >>>>> > >>>>> SELECT * FROM t VERSION AS OF ’tag-name’ > >>>>> > >>>>> to travel to a tag. > >>>>> > >>>>> ## Tag class > >>>>> I agree with you that we can reuse the Snapshot class. We can introduce > >>>>> `TagManager` > >>>>> only to manage tags. > >>>>> > >>>>> ## Expiring Snapshot > >>>>>> why not record it in ManifestEntry? > >>>>> This is because every time Paimon generate a snapshot, it will create > >>>>> new > >>>>> ManifestEntries > >>>>> for data files. Consider this scenario, if we record it in > >>>>> ManifestEntry, > >>>>> assuming we commit > >>>>> data file A to snapshot #1, we will get manifest entry Entry#1 as [ADD, > >>>>> A, commit at #1]. > >>>>> Then we commit -A to snapshot #2, we will get manifest entry Entry#2 as > >>>>> [DELETE, A, ?], > >>>>> as you can see, we cannot know at which snapshot we commit the file A. > >>>>> So > >>>>> we have to > >>>>> record this information to data file meta directly. > >>>>> > >>>>>> We should note that "record it in `DataFileMeta` should be done before > >>>>> “tag” > >>>>> and document version compatibility. > >>>>> > >>>>> I will add message for this. > >>>>> > >>>>> Best, > >>>>> Yu Zelin > >>>>> > >>>>> > >>>>>> 2023年5月29日 10:29，Jingsong Li <[email protected]> 写道： > >>>>>> > >>>>>> Thanks Zelin for the update. > >>>>>> > >>>>>> ## TAG ID > >>>>>> > >>>>>> Is this useful? We have tag-name, snapshot-id, and now introducing a > >>>>>> tag id? What is used? > >>>>>> > >>>>>> ## Time Travel > >>>>>> > >>>>>> SELECT * FROM t VERSION AS OF tag-name.<name> > >>>>>> > >>>>>> This does not look like sql standard. > >>>>>> > >>>>>> Why do we introduce this `tag-name` prefix? > >>>>>> > >>>>>> ## Tag class > >>>>>> > >>>>>> Why not just use the Snapshot class? It looks like we don't need to > >>>>>> introduce Tag class. We can just copy the snapshot file to tag/. > >>>>>> > >>>>>> ## Expiring Snapshot > >>>>>> > >>>>>> We should note that "record it in `DataFileMeta`" should be done > >>>>>> before "tag". And document version compatibility. > >>>>>> And why not record it in ManifestEntry? > >>>>>> > >>>>>> Best, > >>>>>> Jingsong > >>>>>> > >>>>>> On Fri, May 26, 2023 at 11:15 AM yu zelin <[email protected]> > >>>>>> wrote: > >>>>>>> > >>>>>>> Hi, all, > >>>>>>> > >>>>>>> FYI, I have updated the PIP [1]. > >>>>>>> > >>>>>>> Main changes: > >>>>>>> - Use new name `tag` > >>>>>>> - Enrich Motivation > >>>>>>> - New Section `Data Files Handling` to describe how to determine a > >>>>>>> data > >>>>> files can be deleted. > >>>>>>> > >>>>>>> Best, > >>>>>>> Yu Zelin > >>>>>>> > >>>>>>> [1] https://cwiki.apache.org/confluence/x/NxE0Dw > >>>>>>> > >>>>>>>> 2023年5月24日 17:18，yu zelin <[email protected]> 写道： > >>>>>>>> > >>>>>>>> Hi, Guojun, > >>>>>>>> > >>>>>>>> I’d like to share my thoughts about your questions. > >>>>>>>> > >>>>>>>> 1. Expiration of savepoint > >>>>>>>> In my opinion, savepoints are created in a long interval, so there > >>>>> will not exist too many of them. > >>>>>>>> If users create a savepoint per day, there are 365 savepoints a year. > >>>>> So I didn’t consider expiration > >>>>>>>> of it, and I think provide a flink action like `delete-savepoint id = > >>>>> 1` is enough now. > >>>>>>>> But if it is really important, we can introduce table options to do > >>>>> so. I think we can do it like expiring > >>>>>>>> snapshots. > >>>>>>>> > >>>>>>>> 2. > id of compacted snapshot picked by the savepoint > >>>>>>>> My initial idea is picking a compacted snapshot or doing compaction > >>>>> before creating savepoint. But > >>>>>>>> After discuss with Jingsong, I found it’s difficult. So now I suppose > >>>>> to directly create savepoint from > >>>>>>>> the given snapshot. Maybe we can optimize it later. > >>>>>>>> The changes will be updated soon. > >>>>>>>>> manifest file list in system-table > >>>>>>>> I think manifest file is not very important for users. Users can find > >>>>> when a savepoint is created, and > >>>>>>>> get the savepoint id, then they can query it from the savepoint by > >>>>>>>> the > >>>>> id. I did’t see what scenario > >>>>>>>> the users need the manifest file information. What do you think? > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Yu Zelin > >>>>>>>> > >>>>>>>>> 2023年5月24日 10:50，Guojun Li <[email protected]> 写道： > >>>>>>>>> > >>>>>>>>> Thanks zelin for bringing up the discussion. I'm thinking about: > >>>>>>>>> 1. How to manage the savepoints if there are no expiration > >>>>>>>>> mechanism, > >>>>> by > >>>>>>>>> the TTL management of storages or external script? > >>>>>>>>> 2. I think the id of compacted snapshot picked by the savepoint and > >>>>>>>>> manifest file list is also important information for users, could > >>>>> these > >>>>>>>>> information be stored in the system-table? > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Guojun > >>>>>>>>> > >>>>>>>>> On Mon, May 22, 2023 at 9:13 PM Jingsong Li <[email protected]> > >>>>> wrote: > >>>>>>>>> > >>>>>>>>>> FYI > >>>>>>>>>> > >>>>>>>>>> The PIP lacks a table to show Discussion thread & Vote thread & > >>>>> ISSUE... > >>>>>>>>>> > >>>>>>>>>> Best > >>>>>>>>>> Jingsong > >>>>>>>>>> > >>>>>>>>>> On Mon, May 22, 2023 at 4:48 PM yu zelin <[email protected]> > >>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi, all, > >>>>>>>>>>> > >>>>>>>>>>> Thank all of you for your suggestions and questions. After reading > >>>>> your > >>>>>>>>>> suggestions, I adopt some of them and I want to share my opinions > >>>>> here. > >>>>>>>>>>> > >>>>>>>>>>> To make my statements more clear, I will still use the word > >>>>> `savepoint`. > >>>>>>>>>> When we make a consensus, the name may be changed. > >>>>>>>>>>> > >>>>>>>>>>> 1. The purposes of savepoint > >>>>>>>>>>> > >>>>>>>>>>> As Shammon mentioned, Flink and database also have the concept of > >>>>>>>>>> `savepoint`. So it’s better to clarify the purposes of our > >>>>>>>>>> savepoint. > >>>>>>>>>> Thanks for Nicholas and Jingsong, I think your explanations are > >>>>>>>>>> very > >>>>> clear. > >>>>>>>>>> I’d like to give my summary: > >>>>>>>>>>> > >>>>>>>>>>> (1) Fault recovery (or we can say disaster recovery). Users can > >>>>>>>>>>> ROLL > >>>>>>>>>> BACK to a savepoint if needed. If user rollbacks to a savepoint, > >>>>>>>>>> the > >>>>> table > >>>>>>>>>> will hold the data in the savepoint and the data committed after > >>>>>>>>>> the > >>>>>>>>>> savepoint will be deleted. In this scenario we need savepoint > >>>>>>>>>> because > >>>>>>>>>> snapshots may have expired, the savepoint can keep longer and save > >>>>> user’s > >>>>>>>>>> old data. > >>>>>>>>>>> > >>>>>>>>>>> (2) Record versions of data at a longer interval (typically daily > >>>>> level > >>>>>>>>>> or weekly level). With savepoint, user can query the old data in > >>>>> batch > >>>>>>>>>> mode. Comparing to copy records to a new table or merge incremental > >>>>> records > >>>>>>>>>> with old records (like using merge into in Hive), the savepoint is > >>>>> more > >>>>>>>>>> lightweight because we don’t copy data files, we just record the > >>>>> meta data > >>>>>>>>>> of them. > >>>>>>>>>>> > >>>>>>>>>>> As you can see, savepoint is very similar to snapshot. The > >>>>> differences > >>>>>>>>>> are: > >>>>>>>>>>> > >>>>>>>>>>> (1) Savepoint lives longer. In most cases, snapshot’s life time is > >>>>>>>>>> about several minutes to hours. We suppose the savepoint can live > >>>>> several > >>>>>>>>>> days, weeks, or even months. > >>>>>>>>>>> > >>>>>>>>>>> (2) Savepoint is mainly used for batch reading for historical > >>>>>>>>>>> data. > >>>>> In > >>>>>>>>>> this PIP, we don’t introduce streaming reading for savepoint. > >>>>>>>>>>> > >>>>>>>>>>> 2. Candidates of name > >>>>>>>>>>> > >>>>>>>>>>> I agree with Jingsong that we can use a new name. Since the > >>>>>>>>>>> purpose > >>>>> and > >>>>>>>>>> mechanism (savepoint is very similar to snapshot) of savepoint is > >>>>> similar > >>>>>>>>>> to `tag` in iceberg, maybe we can use `tag`. > >>>>>>>>>>> > >>>>>>>>>>> In my opinion, an alternative is `anchor`. All the snapshots are > >>>>> like > >>>>>>>>>> the navigation path of the streaming data, and an `anchor` can stop > >>>>> it in a > >>>>>>>>>> place. > >>>>>>>>>>> > >>>>>>>>>>> 3. Public table operations and options > >>>>>>>>>>> > >>>>>>>>>>> We supposed to expose some operations and table options for user > >>>>>>>>>>> to > >>>>>>>>>> manage the savepoint. > >>>>>>>>>>> > >>>>>>>>>>> (1) Operations (Currently for Flink) > >>>>>>>>>>> We provide flink actions to manage savepoints: > >>>>>>>>>>> create-savepoint: To generate a savepoint from latest snapshot. > >>>>>>>>>> Support to create from specified snapshot. > >>>>>>>>>>> delete-savepoint: To delete specified savepoint. > >>>>>>>>>>> rollback-to: To roll back to a specified savepoint. > >>>>>>>>>>> > >>>>>>>>>>> (2) Table options > >>>>>>>>>>> We suppose to provide options for creating savepoint periodically: > >>>>>>>>>>> savepoint.create-time: When to create the savepoint. Example: > >>>>>>>>>>> 00:00 > >>>>>>>>>>> savepoint.create-interval: Interval between the creation of two > >>>>>>>>>> savepoints. Examples: 2 d. > >>>>>>>>>>> savepoint.time-retained: The maximum time of savepoints to retain. > >>>>>>>>>>> > >>>>>>>>>>> (3) Procedures (future work) > >>>>>>>>>>> Spark supports SQL extension. After we support Spark CALL > >>>>> statement, we > >>>>>>>>>> can provide procedures to create, delete or rollback to savepoint > >>>>> for Spark > >>>>>>>>>> users. > >>>>>>>>>>> > >>>>>>>>>>> Support of CALL is on the road map of Flink. In future version, we > >>>>> can > >>>>>>>>>> also support savepoint-related procedures for Flink users. > >>>>>>>>>>> > >>>>>>>>>>> 4. Expiration of data files > >>>>>>>>>>> > >>>>>>>>>>> Currently, when a snapshot is expired, data files that not be used > >>>>> by > >>>>>>>>>> other snapshots. After we introduce the savepoint, we must make > >>>>>>>>>> sure > >>>>> the > >>>>>>>>>> data files saved by savepoint will not be deleted. > >>>>>>>>>>> > >>>>>>>>>>> Conversely, when a savepoint is deleted, the data files that not > >>>>>>>>>>> be > >>>>>>>>>> used by existing snapshots and other savepoints will be deleted. > >>>>>>>>>>> > >>>>>>>>>>> I have wrote some POC codes to implement it. I will update the > >>>>> mechanism > >>>>>>>>>> in PIP soon. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Yu Zelin > >>>>>>>>>>> > >>>>>>>>>>>> 2023年5月21日 20:54，Jingsong Li <[email protected]> 写道： > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks Yun for your information. > >>>>>>>>>>>> > >>>>>>>>>>>> We need to be careful to avoid confusion between Paimon and Flink > >>>>>>>>>>>> concepts about "savepoint" > >>>>>>>>>>>> > >>>>>>>>>>>> Maybe we don't have to insist on using this "savepoint", for > >>>>> example, > >>>>>>>>>>>> TAG is also a candidate just like Iceberg [1] > >>>>>>>>>>>> > >>>>>>>>>>>> [1] https://iceberg.apache.org/docs/latest/branching/ > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> Jingsong > >>>>>>>>>>>> > >>>>>>>>>>>> On Sun, May 21, 2023 at 8:51 PM Jingsong Li < > >>>>> [email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks Nicholas for your detailed requirements. > >>>>>>>>>>>>> > >>>>>>>>>>>>> We need to supplement user requirements in FLIP, which is mainly > >>>>> aimed > >>>>>>>>>>>>> at two purposes: > >>>>>>>>>>>>> 1. Fault recovery for data errors (named: restore or > >>>>>>>>>>>>> rollback-to) > >>>>>>>>>>>>> 2. Used to record versions at the day level (such as), targeting > >>>>>>>>>> batch queries > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best, > >>>>>>>>>>>>> Jingsong > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Sat, May 20, 2023 at 2:55 PM Yun Tang <[email protected]> > >>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Guys, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Since we use Paimon with Flink in most cases, I think we need > >>>>>>>>>>>>>> to > >>>>>>>>>> identify the same word "savepoint" in different systems. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> For Flink, savepoint means: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. Triggered by users, not periodically triggered by the > >>>>>>>>>>>>>> system > >>>>>>>>>> itself. However, this FLIP wants to support it created > >>>>>>>>>> periodically. > >>>>>>>>>>>>>> 2. Even the so-called incremental native savepoint [1], it > >>>>>>>>>>>>>> will > >>>>>>>>>> not depend on the previous checkpoints or savepoints, it will still > >>>>> copy > >>>>>>>>>> files on DFS to the self-contained savepoint folder. However, from > >>>>> the > >>>>>>>>>> description of this FLIP about the deletion of expired snapshot > >>>>> files, > >>>>>>>>>> paimion savepoint will refer to the previously existing files > >>>>> directly. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I don't think we need to make the semantics of Paimon totally > >>>>>>>>>>>>>> the > >>>>>>>>>> same as Flink's. However, we need to introduce a table to tell the > >>>>>>>>>> difference compared with Flink and discuss about the difference. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [1] > >>>>>>>>>> > >>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-203%3A+Incremental+savepoints#FLIP203:Incrementalsavepoints-Semantic > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Best > >>>>>>>>>>>>>> Yun Tang > >>>>>>>>>>>>>> ________________________________ > >>>>>>>>>>>>>> From: Nicholas Jiang <[email protected]> > >>>>>>>>>>>>>> Sent: Friday, May 19, 2023 17:40 > >>>>>>>>>>>>>> To: [email protected] <[email protected]> > >>>>>>>>>>>>>> Subject: Re: [DISCUSS] PIP-4 Support savepoint > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Guys, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks Zelin for driving the savepoint proposal. I propose some > >>>>>>>>>> opinions for savepont: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- About "introduce savepoint for Paimon to persist full data > >>>>>>>>>>>>>> in > >>>>> a > >>>>>>>>>> time point" > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The motivation of savepoint proposal is more like snapshot TTL > >>>>>>>>>> management. Actually, disaster recovery is very much mission > >>>>> critical for > >>>>>>>>>> any software. Especially when it comes to data systems, the impact > >>>>> could be > >>>>>>>>>> very serious leading to delay in business decisions or even wrong > >>>>> business > >>>>>>>>>> decisions at times. Savepoint is proposed to assist users in > >>>>> recovering > >>>>>>>>>> data from a previous state: "savepoint" and "restore". > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> "savepoint" saves the Paimon table as of the commit time, > >>>>> therefore > >>>>>>>>>> if there is a savepoint, the data generated in the corresponding > >>>>> commit > >>>>>>>>>> could not be clean. Meanwhile, savepoint could let user restore the > >>>>> table > >>>>>>>>>> to this savepoint at a later point in time if need be. On similar > >>>>> lines, > >>>>>>>>>> savepoint cannot be triggered on a commit that is already cleaned > >>>>>>>>>> up. > >>>>>>>>>> Savepoint is synonymous to taking a backup, just that we don't make > >>>>> a new > >>>>>>>>>> copy of the table, but just save the state of the table elegantly > >>>>>>>>>> so > >>>>> that > >>>>>>>>>> we can restore it later when in need. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> "restore" lets you restore your table to one of the savepoint > >>>>>>>>>> commit. Meanwhile, it cannot be undone (or reversed) and so care > >>>>> should be > >>>>>>>>>> taken before doing a restore. At this time, Paimon would delete all > >>>>> data > >>>>>>>>>> files and commit files (timeline files) greater than the savepoint > >>>>> commit > >>>>>>>>>> to which the table is being restored. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> BTW, it's better to introduce snapshot view based on savepoint, > >>>>>>>>>> which could improve query performance of historical data for Paimon > >>>>> table. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- About Public API of savepont > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Current introduced savepoint interfaces in Public API are not > >>>>> enough > >>>>>>>>>> for users, for example, deleteSavepoint, restoreSavepoint etc. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- About "Paimon's savepoint need to be combined with Flink's > >>>>>>>>>> savepoint": > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> If paimon supports savepoint mechanism and provides savepoint > >>>>>>>>>> interfaces, the integration with Flink's savepoint is not blocked > >>>>> for this > >>>>>>>>>> proposal. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In summary, savepoint is not only used to improve the query > >>>>>>>>>> performance of historical data, but also used for disaster recovery > >>>>>>>>>> processing. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2023/05/17 09:53:11 Jingsong Li wrote: > >>>>>>>>>>>>>>> What Shammon mentioned is interesting. I agree with what he > >>>>>>>>>>>>>>> said > >>>>>>>>>> about > >>>>>>>>>>>>>>> the differences in savepoints between databases and stream > >>>>>>>>>> computing. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> About "Paimon's savepoint need to be combined with Flink's > >>>>>>>>>> savepoint": > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I think it is possible, but we may need to deal with this in > >>>>> another > >>>>>>>>>>>>>>> mechanism, because the snapshots after savepoint may expire. > >>>>>>>>>>>>>>> We > >>>>> need > >>>>>>>>>>>>>>> to compare data between two savepoints to generate incremental > >>>>> data > >>>>>>>>>>>>>>> for streaming read. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> But this may not need to block FLIP, it looks like the current > >>>>>>>>>> design > >>>>>>>>>>>>>>> does not break the future combination? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>> Jingsong > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wed, May 17, 2023 at 5:33 PM Shammon FY <[email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Caizhi, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks for your comments. As you mentioned, I think we may > >>>>> need to > >>>>>>>>>> discuss > >>>>>>>>>>>>>>>> the role of savepoint in Paimon. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> If I understand correctly, the main feature of savepoint in > >>>>>>>>>>>>>>>> the > >>>>>>>>>> current PIP > >>>>>>>>>>>>>>>> is that the savepoint will not be expired, and users can > >>>>> perform a > >>>>>>>>>> query on > >>>>>>>>>>>>>>>> the savepoint according to time-travel. Besides that, there > >>>>>>>>>>>>>>>> is > >>>>>>>>>> savepoint in > >>>>>>>>>>>>>>>> the database and Flink. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 1. Savepoint in database. The database can roll back table > >>>>> data to > >>>>>>>>>> the > >>>>>>>>>>>>>>>> specified 'version' based on savepoint. So the key point of > >>>>>>>>>> savepoint in > >>>>>>>>>>>>>>>> the database is to rollback data. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 2. Savepoint in Flink. Users can trigger a savepoint with a > >>>>>>>>>> specific > >>>>>>>>>>>>>>>> 'path', and save all data of state to the savepoint for job. > >>>>> Then > >>>>>>>>>> users can > >>>>>>>>>>>>>>>> create a new job based on the savepoint to continue consuming > >>>>>>>>>> incremental > >>>>>>>>>>>>>>>> data. I think the core capabilities are: backup for a job, > >>>>>>>>>>>>>>>> and > >>>>>>>>>> resume a job > >>>>>>>>>>>>>>>> based on the savepoint. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> In addition to the above, Paimon may also face data write > >>>>>>>>>> corruption and > >>>>>>>>>>>>>>>> need to recover data based on the specified savepoint. So we > >>>>> may > >>>>>>>>>> need to > >>>>>>>>>>>>>>>> consider what abilities should Paimon savepoint need besides > >>>>> the > >>>>>>>>>> ones > >>>>>>>>>>>>>>>> mentioned in the current PIP? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Additionally, as mentioned above, Flink also has > >>>>>>>>>>>>>>>> savepoint mechanism. During the process of streaming data > >>>>>>>>>>>>>>>> from > >>>>>>>>>> Flink to > >>>>>>>>>>>>>>>> Paimon, does Paimon's savepoint need to be combined with > >>>>> Flink's > >>>>>>>>>> savepoint? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>> Shammon FY > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Wed, May 17, 2023 at 4:02 PM Caizhi Weng < > >>>>> [email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi developers! > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks Zelin for bringing up the discussion. The proposal > >>>>> seems > >>>>>>>>>> good to me > >>>>>>>>>>>>>>>>> overall. However I'd also like to bring up a few options. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1. As Jingsong mentioned, Savepoint class should not become > >>>>>>>>>>>>>>>>> a > >>>>>>>>>> public API, > >>>>>>>>>>>>>>>>> at least for now. What we need to discuss for the public API > >>>>> is > >>>>>>>>>> how the > >>>>>>>>>>>>>>>>> users can create or delete savepoints. For example, what the > >>>>>>>>>> table option > >>>>>>>>>>>>>>>>> looks like, what commands and options are provided for the > >>>>> Flink > >>>>>>>>>> action, > >>>>>>>>>>>>>>>>> etc. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2. Currently most Flink actions are related to streaming > >>>>>>>>>> processing, so > >>>>>>>>>>>>>>>>> only Flink can support them. However, savepoint creation and > >>>>>>>>>> deletion seems > >>>>>>>>>>>>>>>>> like a feature for batch processing. So aside from Flink > >>>>> actions, > >>>>>>>>>> shall we > >>>>>>>>>>>>>>>>> also provide something like Spark actions for savepoints? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I would also like to comment on Shammon's views. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Should we introduce an option for savepoint path which may > >>>>>>>>>>>>>>>>> be > >>>>>>>>>> different > >>>>>>>>>>>>>>>>>> from 'warehouse'? Then users can backup the data of > >>>>> savepoint. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I don't see this is necessary. To backup a table the user > >>>>>>>>>>>>>>>>> just > >>>>>>>>>> need to copy > >>>>>>>>>>>>>>>>> all files from the table directory. Savepoint in Paimon, as > >>>>> far > >>>>>>>>>> as I > >>>>>>>>>>>>>>>>> understand, is mainly for users to review historical data, > >>>>>>>>>>>>>>>>> not > >>>>>>>>>> for backing > >>>>>>>>>>>>>>>>> up tables. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Will the savepoint copy data files from snapshot or only > >>>>>>>>>>>>>>>>> save > >>>>>>>>>> meta files? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> It would be a heavy burden if a savepoint copies all its > >>>>> files. > >>>>>>>>>> As I > >>>>>>>>>>>>>>>>> mentioned above, savepoint is not for backing up tables. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> How can users create a new table and restore data from the > >>>>>>>>>> specified > >>>>>>>>>>>>>>>>>> savepoint? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> This reminds me of savepoints in Flink. Still, savepoint is > >>>>> not > >>>>>>>>>> for backing > >>>>>>>>>>>>>>>>> up tables so I guess we don't need to support "restoring > >>>>>>>>>>>>>>>>> data" > >>>>>>>>>> from a > >>>>>>>>>>>>>>>>> savepoint. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Shammon FY <[email protected]> 于2023年5月17日周三 10:32写道： > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks Zelin for initiating this discussion. I have some > >>>>>>>>>> comments: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 1. Should we introduce an option for savepoint path which > >>>>> may be > >>>>>>>>>>>>>>>>> different > >>>>>>>>>>>>>>>>>> from 'warehouse'? Then users can backup the data of > >>>>> savepoint. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 2. Will the savepoint copy data files from snapshot or only > >>>>> save > >>>>>>>>>> meta > >>>>>>>>>>>>>>>>>> files? The description in the PIP "After we introduce > >>>>> savepoint, > >>>>>>>>>> we > >>>>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>> also check if the data files are used by savepoints." looks > >>>>> like > >>>>>>>>>> we only > >>>>>>>>>>>>>>>>>> save meta files for savepoint. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 3. How can users create a new table and restore data from > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>> specified > >>>>>>>>>>>>>>>>>> savepoint? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>> Shammon FY > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Wed, May 17, 2023 at 10:19 AM Jingsong Li < > >>>>>>>>>> [email protected]> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks Zelin for driving. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Some comments: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 1. I think it's possible to advance `Proposed Changes` to > >>>>> the > >>>>>>>>>> top, > >>>>>>>>>>>>>>>>>>> Public API has no meaning if I don't know how to do it. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 2. Public API, Savepoint and SavepointManager are not > >>>>>>>>>>>>>>>>>>> Public > >>>>>>>>>> API, only > >>>>>>>>>>>>>>>>>>> Flink action or configuration option should be public API. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 3.Maybe we can have a separate chapter to describe > >>>>>>>>>>>>>>>>>>> `savepoint.create-interval`, maybe 'Periodically > >>>>> savepoint'? It > >>>>>>>>>> is not > >>>>>>>>>>>>>>>>>>> just an interval, because the true user case is savepoint > >>>>> after > >>>>>>>>>> 0:00. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 4.About 'Interaction with Snapshot', to be continued ... > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>> Jingsong > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Tue, May 16, 2023 at 7:07 PM yu zelin < > >>>>> [email protected] > >>>>>>>>>>> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi, Paimon Devs, > >>>>>>>>>>>>>>>>>>>> I’d like to start a discussion about PIP-4[1]. In this > >>>>>>>>>> PIP, I > >>>>>>>>>>>>>>>>> want > >>>>>>>>>>>>>>>>>>> to talk about why we need savepoint, and some thoughts > >>>>>>>>>>>>>>>>>>> about > >>>>>>>>>> managing > >>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> using savepoint. Look forward to your question and > >>>>> suggestions. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>> Yu Zelin > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/confluence/x/NxE0Dw > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>> > > >
