I think we can just throw exceptions for pure numeric tag names. Iceberg's behavior looks confusing.
Best, Jingsong On Tue, May 30, 2023 at 3:40 PM yu zelin <[email protected]> wrote: > > Hi, Shammon, > > An intuitive way is use numeric string to indicate snapshot and non-numeric > string to indicate tag. > For example: > > SELECT * FROM t VERSION AS OF 1 —to snapshot #1 > SELECT * FROM t VERSION AS OF ‘last_year’ —to tag `last_year` > > This is also how iceberg do [1]. > > However, if we use this way, the tag name cannot be numeric string. I think > this is acceptable and I will add this to the document. > > Best, > Yu Zelin > > [1] https://iceberg.apache.org/docs/latest/spark-queries/#sql > > > 2023年5月30日 12:17，Shammon FY <[email protected]> 写道： > > > > Hi zelin, > > > > Thanks for your update. I have one comment about Time Travel on savepoint. > > > > Currently we can use statement in spark for specific snapshot 1 > > SELECT * FROM t VERSION AS OF 1; > > > > My point is how can we distinguish between snapshot and savepoint when > > users submit a statement as followed: > > SELECT * FROM t VERSION AS OF <version value>; > > > > Best, > > Shammon FY > > > > On Tue, May 30, 2023 at 11:37 AM yu zelin <[email protected]> wrote: > > > >> Hi, Jingsong, > >> > >> Thanks for your feedback. > >> > >> ## TAG ID > >> It seems the id is useless currently. I’ll remove it. > >> > >> ## Time Travel Syntax > >> Since tag id is removed, we can just use: > >> > >> SELECT * FROM t VERSION AS OF ’tag-name’ > >> > >> to travel to a tag. > >> > >> ## Tag class > >> I agree with you that we can reuse the Snapshot class. We can introduce > >> `TagManager` > >> only to manage tags. > >> > >> ## Expiring Snapshot > >>> why not record it in ManifestEntry? > >> This is because every time Paimon generate a snapshot, it will create new > >> ManifestEntries > >> for data files. Consider this scenario, if we record it in ManifestEntry, > >> assuming we commit > >> data file A to snapshot #1, we will get manifest entry Entry#1 as [ADD, > >> A, commit at #1]. > >> Then we commit -A to snapshot #2, we will get manifest entry Entry#2 as > >> [DELETE, A, ?], > >> as you can see, we cannot know at which snapshot we commit the file A. So > >> we have to > >> record this information to data file meta directly. > >> > >>> We should note that "record it in `DataFileMeta` should be done before > >> “tag” > >> and document version compatibility. > >> > >> I will add message for this. > >> > >> Best, > >> Yu Zelin > >> > >> > >>> 2023年5月29日 10:29，Jingsong Li <[email protected]> 写道： > >>> > >>> Thanks Zelin for the update. > >>> > >>> ## TAG ID > >>> > >>> Is this useful? We have tag-name, snapshot-id, and now introducing a > >>> tag id? What is used? > >>> > >>> ## Time Travel > >>> > >>> SELECT * FROM t VERSION AS OF tag-name.<name> > >>> > >>> This does not look like sql standard. > >>> > >>> Why do we introduce this `tag-name` prefix? > >>> > >>> ## Tag class > >>> > >>> Why not just use the Snapshot class? It looks like we don't need to > >>> introduce Tag class. We can just copy the snapshot file to tag/. > >>> > >>> ## Expiring Snapshot > >>> > >>> We should note that "record it in `DataFileMeta`" should be done > >>> before "tag". And document version compatibility. > >>> And why not record it in ManifestEntry? > >>> > >>> Best, > >>> Jingsong > >>> > >>> On Fri, May 26, 2023 at 11:15 AM yu zelin <[email protected]> wrote: > >>>> > >>>> Hi, all, > >>>> > >>>> FYI, I have updated the PIP [1]. > >>>> > >>>> Main changes: > >>>> - Use new name `tag` > >>>> - Enrich Motivation > >>>> - New Section `Data Files Handling` to describe how to determine a data > >> files can be deleted. > >>>> > >>>> Best, > >>>> Yu Zelin > >>>> > >>>> [1] https://cwiki.apache.org/confluence/x/NxE0Dw > >>>> > >>>>> 2023年5月24日 17:18，yu zelin <[email protected]> 写道： > >>>>> > >>>>> Hi, Guojun, > >>>>> > >>>>> I’d like to share my thoughts about your questions. > >>>>> > >>>>> 1. Expiration of savepoint > >>>>> In my opinion, savepoints are created in a long interval, so there > >> will not exist too many of them. > >>>>> If users create a savepoint per day, there are 365 savepoints a year. > >> So I didn’t consider expiration > >>>>> of it, and I think provide a flink action like `delete-savepoint id = > >> 1` is enough now. > >>>>> But if it is really important, we can introduce table options to do > >> so. I think we can do it like expiring > >>>>> snapshots. > >>>>> > >>>>> 2. > id of compacted snapshot picked by the savepoint > >>>>> My initial idea is picking a compacted snapshot or doing compaction > >> before creating savepoint. But > >>>>> After discuss with Jingsong, I found it’s difficult. So now I suppose > >> to directly create savepoint from > >>>>> the given snapshot. Maybe we can optimize it later. > >>>>> The changes will be updated soon. > >>>>>> manifest file list in system-table > >>>>> I think manifest file is not very important for users. Users can find > >> when a savepoint is created, and > >>>>> get the savepoint id, then they can query it from the savepoint by the > >> id. I did’t see what scenario > >>>>> the users need the manifest file information. What do you think? > >>>>> > >>>>> Best, > >>>>> Yu Zelin > >>>>> > >>>>>> 2023年5月24日 10:50，Guojun Li <[email protected]> 写道： > >>>>>> > >>>>>> Thanks zelin for bringing up the discussion. I'm thinking about: > >>>>>> 1. How to manage the savepoints if there are no expiration mechanism, > >> by > >>>>>> the TTL management of storages or external script? > >>>>>> 2. I think the id of compacted snapshot picked by the savepoint and > >>>>>> manifest file list is also important information for users, could > >> these > >>>>>> information be stored in the system-table? > >>>>>> > >>>>>> Best, > >>>>>> Guojun > >>>>>> > >>>>>> On Mon, May 22, 2023 at 9:13 PM Jingsong Li <[email protected]> > >> wrote: > >>>>>> > >>>>>>> FYI > >>>>>>> > >>>>>>> The PIP lacks a table to show Discussion thread & Vote thread & > >> ISSUE... > >>>>>>> > >>>>>>> Best > >>>>>>> Jingsong > >>>>>>> > >>>>>>> On Mon, May 22, 2023 at 4:48 PM yu zelin <[email protected]> > >> wrote: > >>>>>>>> > >>>>>>>> Hi, all, > >>>>>>>> > >>>>>>>> Thank all of you for your suggestions and questions. After reading > >> your > >>>>>>> suggestions, I adopt some of them and I want to share my opinions > >> here. > >>>>>>>> > >>>>>>>> To make my statements more clear, I will still use the word > >> `savepoint`. > >>>>>>> When we make a consensus, the name may be changed. > >>>>>>>> > >>>>>>>> 1. The purposes of savepoint > >>>>>>>> > >>>>>>>> As Shammon mentioned, Flink and database also have the concept of > >>>>>>> `savepoint`. So it’s better to clarify the purposes of our savepoint. > >>>>>>> Thanks for Nicholas and Jingsong, I think your explanations are very > >> clear. > >>>>>>> I’d like to give my summary: > >>>>>>>> > >>>>>>>> (1) Fault recovery (or we can say disaster recovery). Users can ROLL > >>>>>>> BACK to a savepoint if needed. If user rollbacks to a savepoint, the > >> table > >>>>>>> will hold the data in the savepoint and the data committed after the > >>>>>>> savepoint will be deleted. In this scenario we need savepoint because > >>>>>>> snapshots may have expired, the savepoint can keep longer and save > >> user’s > >>>>>>> old data. > >>>>>>>> > >>>>>>>> (2) Record versions of data at a longer interval (typically daily > >> level > >>>>>>> or weekly level). With savepoint, user can query the old data in > >> batch > >>>>>>> mode. Comparing to copy records to a new table or merge incremental > >> records > >>>>>>> with old records (like using merge into in Hive), the savepoint is > >> more > >>>>>>> lightweight because we don’t copy data files, we just record the > >> meta data > >>>>>>> of them. > >>>>>>>> > >>>>>>>> As you can see, savepoint is very similar to snapshot. The > >> differences > >>>>>>> are: > >>>>>>>> > >>>>>>>> (1) Savepoint lives longer. In most cases, snapshot’s life time is > >>>>>>> about several minutes to hours. We suppose the savepoint can live > >> several > >>>>>>> days, weeks, or even months. > >>>>>>>> > >>>>>>>> (2) Savepoint is mainly used for batch reading for historical data. > >> In > >>>>>>> this PIP, we don’t introduce streaming reading for savepoint. > >>>>>>>> > >>>>>>>> 2. Candidates of name > >>>>>>>> > >>>>>>>> I agree with Jingsong that we can use a new name. Since the purpose > >> and > >>>>>>> mechanism (savepoint is very similar to snapshot) of savepoint is > >> similar > >>>>>>> to `tag` in iceberg, maybe we can use `tag`. > >>>>>>>> > >>>>>>>> In my opinion, an alternative is `anchor`. All the snapshots are > >> like > >>>>>>> the navigation path of the streaming data, and an `anchor` can stop > >> it in a > >>>>>>> place. > >>>>>>>> > >>>>>>>> 3. Public table operations and options > >>>>>>>> > >>>>>>>> We supposed to expose some operations and table options for user to > >>>>>>> manage the savepoint. > >>>>>>>> > >>>>>>>> (1) Operations (Currently for Flink) > >>>>>>>> We provide flink actions to manage savepoints: > >>>>>>>> create-savepoint: To generate a savepoint from latest snapshot. > >>>>>>> Support to create from specified snapshot. > >>>>>>>> delete-savepoint: To delete specified savepoint. > >>>>>>>> rollback-to: To roll back to a specified savepoint. > >>>>>>>> > >>>>>>>> (2) Table options > >>>>>>>> We suppose to provide options for creating savepoint periodically: > >>>>>>>> savepoint.create-time: When to create the savepoint. Example: 00:00 > >>>>>>>> savepoint.create-interval: Interval between the creation of two > >>>>>>> savepoints. Examples: 2 d. > >>>>>>>> savepoint.time-retained: The maximum time of savepoints to retain. > >>>>>>>> > >>>>>>>> (3) Procedures (future work) > >>>>>>>> Spark supports SQL extension. After we support Spark CALL > >> statement, we > >>>>>>> can provide procedures to create, delete or rollback to savepoint > >> for Spark > >>>>>>> users. > >>>>>>>> > >>>>>>>> Support of CALL is on the road map of Flink. In future version, we > >> can > >>>>>>> also support savepoint-related procedures for Flink users. > >>>>>>>> > >>>>>>>> 4. Expiration of data files > >>>>>>>> > >>>>>>>> Currently, when a snapshot is expired, data files that not be used > >> by > >>>>>>> other snapshots. After we introduce the savepoint, we must make sure > >> the > >>>>>>> data files saved by savepoint will not be deleted. > >>>>>>>> > >>>>>>>> Conversely, when a savepoint is deleted, the data files that not be > >>>>>>> used by existing snapshots and other savepoints will be deleted. > >>>>>>>> > >>>>>>>> I have wrote some POC codes to implement it. I will update the > >> mechanism > >>>>>>> in PIP soon. > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Yu Zelin > >>>>>>>> > >>>>>>>>> 2023年5月21日 20:54，Jingsong Li <[email protected]> 写道： > >>>>>>>>> > >>>>>>>>> Thanks Yun for your information. > >>>>>>>>> > >>>>>>>>> We need to be careful to avoid confusion between Paimon and Flink > >>>>>>>>> concepts about "savepoint" > >>>>>>>>> > >>>>>>>>> Maybe we don't have to insist on using this "savepoint", for > >> example, > >>>>>>>>> TAG is also a candidate just like Iceberg [1] > >>>>>>>>> > >>>>>>>>> [1] https://iceberg.apache.org/docs/latest/branching/ > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Jingsong > >>>>>>>>> > >>>>>>>>> On Sun, May 21, 2023 at 8:51 PM Jingsong Li < > >> [email protected]> > >>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Thanks Nicholas for your detailed requirements. > >>>>>>>>>> > >>>>>>>>>> We need to supplement user requirements in FLIP, which is mainly > >> aimed > >>>>>>>>>> at two purposes: > >>>>>>>>>> 1. Fault recovery for data errors (named: restore or rollback-to) > >>>>>>>>>> 2. Used to record versions at the day level (such as), targeting > >>>>>>> batch queries > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Jingsong > >>>>>>>>>> > >>>>>>>>>> On Sat, May 20, 2023 at 2:55 PM Yun Tang <[email protected]> > >> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi Guys, > >>>>>>>>>>> > >>>>>>>>>>> Since we use Paimon with Flink in most cases, I think we need to > >>>>>>> identify the same word "savepoint" in different systems. > >>>>>>>>>>> > >>>>>>>>>>> For Flink, savepoint means: > >>>>>>>>>>> > >>>>>>>>>>> 1. Triggered by users, not periodically triggered by the system > >>>>>>> itself. However, this FLIP wants to support it created periodically. > >>>>>>>>>>> 2. Even the so-called incremental native savepoint [1], it will > >>>>>>> not depend on the previous checkpoints or savepoints, it will still > >> copy > >>>>>>> files on DFS to the self-contained savepoint folder. However, from > >> the > >>>>>>> description of this FLIP about the deletion of expired snapshot > >> files, > >>>>>>> paimion savepoint will refer to the previously existing files > >> directly. > >>>>>>>>>>> > >>>>>>>>>>> I don't think we need to make the semantics of Paimon totally the > >>>>>>> same as Flink's. However, we need to introduce a table to tell the > >>>>>>> difference compared with Flink and discuss about the difference. > >>>>>>>>>>> > >>>>>>>>>>> [1] > >>>>>>> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-203%3A+Incremental+savepoints#FLIP203:Incrementalsavepoints-Semantic > >>>>>>>>>>> > >>>>>>>>>>> Best > >>>>>>>>>>> Yun Tang > >>>>>>>>>>> ________________________________ > >>>>>>>>>>> From: Nicholas Jiang <[email protected]> > >>>>>>>>>>> Sent: Friday, May 19, 2023 17:40 > >>>>>>>>>>> To: [email protected] <[email protected]> > >>>>>>>>>>> Subject: Re: [DISCUSS] PIP-4 Support savepoint > >>>>>>>>>>> > >>>>>>>>>>> Hi Guys, > >>>>>>>>>>> > >>>>>>>>>>> Thanks Zelin for driving the savepoint proposal. I propose some > >>>>>>> opinions for savepont: > >>>>>>>>>>> > >>>>>>>>>>> -- About "introduce savepoint for Paimon to persist full data in > >> a > >>>>>>> time point" > >>>>>>>>>>> > >>>>>>>>>>> The motivation of savepoint proposal is more like snapshot TTL > >>>>>>> management. Actually, disaster recovery is very much mission > >> critical for > >>>>>>> any software. Especially when it comes to data systems, the impact > >> could be > >>>>>>> very serious leading to delay in business decisions or even wrong > >> business > >>>>>>> decisions at times. Savepoint is proposed to assist users in > >> recovering > >>>>>>> data from a previous state: "savepoint" and "restore". > >>>>>>>>>>> > >>>>>>>>>>> "savepoint" saves the Paimon table as of the commit time, > >> therefore > >>>>>>> if there is a savepoint, the data generated in the corresponding > >> commit > >>>>>>> could not be clean. Meanwhile, savepoint could let user restore the > >> table > >>>>>>> to this savepoint at a later point in time if need be. On similar > >> lines, > >>>>>>> savepoint cannot be triggered on a commit that is already cleaned up. > >>>>>>> Savepoint is synonymous to taking a backup, just that we don't make > >> a new > >>>>>>> copy of the table, but just save the state of the table elegantly so > >> that > >>>>>>> we can restore it later when in need. > >>>>>>>>>>> > >>>>>>>>>>> "restore" lets you restore your table to one of the savepoint > >>>>>>> commit. Meanwhile, it cannot be undone (or reversed) and so care > >> should be > >>>>>>> taken before doing a restore. At this time, Paimon would delete all > >> data > >>>>>>> files and commit files (timeline files) greater than the savepoint > >> commit > >>>>>>> to which the table is being restored. > >>>>>>>>>>> > >>>>>>>>>>> BTW, it's better to introduce snapshot view based on savepoint, > >>>>>>> which could improve query performance of historical data for Paimon > >> table. > >>>>>>>>>>> > >>>>>>>>>>> -- About Public API of savepont > >>>>>>>>>>> > >>>>>>>>>>> Current introduced savepoint interfaces in Public API are not > >> enough > >>>>>>> for users, for example, deleteSavepoint, restoreSavepoint etc. > >>>>>>>>>>> > >>>>>>>>>>> -- About "Paimon's savepoint need to be combined with Flink's > >>>>>>> savepoint": > >>>>>>>>>>> > >>>>>>>>>>> If paimon supports savepoint mechanism and provides savepoint > >>>>>>> interfaces, the integration with Flink's savepoint is not blocked > >> for this > >>>>>>> proposal. > >>>>>>>>>>> > >>>>>>>>>>> In summary, savepoint is not only used to improve the query > >>>>>>> performance of historical data, but also used for disaster recovery > >>>>>>> processing. > >>>>>>>>>>> > >>>>>>>>>>> On 2023/05/17 09:53:11 Jingsong Li wrote: > >>>>>>>>>>>> What Shammon mentioned is interesting. I agree with what he said > >>>>>>> about > >>>>>>>>>>>> the differences in savepoints between databases and stream > >>>>>>> computing. > >>>>>>>>>>>> > >>>>>>>>>>>> About "Paimon's savepoint need to be combined with Flink's > >>>>>>> savepoint": > >>>>>>>>>>>> > >>>>>>>>>>>> I think it is possible, but we may need to deal with this in > >> another > >>>>>>>>>>>> mechanism, because the snapshots after savepoint may expire. We > >> need > >>>>>>>>>>>> to compare data between two savepoints to generate incremental > >> data > >>>>>>>>>>>> for streaming read. > >>>>>>>>>>>> > >>>>>>>>>>>> But this may not need to block FLIP, it looks like the current > >>>>>>> design > >>>>>>>>>>>> does not break the future combination? > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> Jingsong > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, May 17, 2023 at 5:33 PM Shammon FY <[email protected]> > >>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi Caizhi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for your comments. As you mentioned, I think we may > >> need to > >>>>>>> discuss > >>>>>>>>>>>>> the role of savepoint in Paimon. > >>>>>>>>>>>>> > >>>>>>>>>>>>> If I understand correctly, the main feature of savepoint in the > >>>>>>> current PIP > >>>>>>>>>>>>> is that the savepoint will not be expired, and users can > >> perform a > >>>>>>> query on > >>>>>>>>>>>>> the savepoint according to time-travel. Besides that, there is > >>>>>>> savepoint in > >>>>>>>>>>>>> the database and Flink. > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1. Savepoint in database. The database can roll back table > >> data to > >>>>>>> the > >>>>>>>>>>>>> specified 'version' based on savepoint. So the key point of > >>>>>>> savepoint in > >>>>>>>>>>>>> the database is to rollback data. > >>>>>>>>>>>>> > >>>>>>>>>>>>> 2. Savepoint in Flink. Users can trigger a savepoint with a > >>>>>>> specific > >>>>>>>>>>>>> 'path', and save all data of state to the savepoint for job. > >> Then > >>>>>>> users can > >>>>>>>>>>>>> create a new job based on the savepoint to continue consuming > >>>>>>> incremental > >>>>>>>>>>>>> data. I think the core capabilities are: backup for a job, and > >>>>>>> resume a job > >>>>>>>>>>>>> based on the savepoint. > >>>>>>>>>>>>> > >>>>>>>>>>>>> In addition to the above, Paimon may also face data write > >>>>>>> corruption and > >>>>>>>>>>>>> need to recover data based on the specified savepoint. So we > >> may > >>>>>>> need to > >>>>>>>>>>>>> consider what abilities should Paimon savepoint need besides > >> the > >>>>>>> ones > >>>>>>>>>>>>> mentioned in the current PIP? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Additionally, as mentioned above, Flink also has > >>>>>>>>>>>>> savepoint mechanism. During the process of streaming data from > >>>>>>> Flink to > >>>>>>>>>>>>> Paimon, does Paimon's savepoint need to be combined with > >> Flink's > >>>>>>> savepoint? > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best, > >>>>>>>>>>>>> Shammon FY > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Wed, May 17, 2023 at 4:02 PM Caizhi Weng < > >> [email protected]> > >>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi developers! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks Zelin for bringing up the discussion. The proposal > >> seems > >>>>>>> good to me > >>>>>>>>>>>>>> overall. However I'd also like to bring up a few options. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. As Jingsong mentioned, Savepoint class should not become a > >>>>>>> public API, > >>>>>>>>>>>>>> at least for now. What we need to discuss for the public API > >> is > >>>>>>> how the > >>>>>>>>>>>>>> users can create or delete savepoints. For example, what the > >>>>>>> table option > >>>>>>>>>>>>>> looks like, what commands and options are provided for the > >> Flink > >>>>>>> action, > >>>>>>>>>>>>>> etc. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2. Currently most Flink actions are related to streaming > >>>>>>> processing, so > >>>>>>>>>>>>>> only Flink can support them. However, savepoint creation and > >>>>>>> deletion seems > >>>>>>>>>>>>>> like a feature for batch processing. So aside from Flink > >> actions, > >>>>>>> shall we > >>>>>>>>>>>>>> also provide something like Spark actions for savepoints? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I would also like to comment on Shammon's views. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Should we introduce an option for savepoint path which may be > >>>>>>> different > >>>>>>>>>>>>>>> from 'warehouse'? Then users can backup the data of > >> savepoint. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I don't see this is necessary. To backup a table the user just > >>>>>>> need to copy > >>>>>>>>>>>>>> all files from the table directory. Savepoint in Paimon, as > >> far > >>>>>>> as I > >>>>>>>>>>>>>> understand, is mainly for users to review historical data, not > >>>>>>> for backing > >>>>>>>>>>>>>> up tables. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Will the savepoint copy data files from snapshot or only save > >>>>>>> meta files? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> It would be a heavy burden if a savepoint copies all its > >> files. > >>>>>>> As I > >>>>>>>>>>>>>> mentioned above, savepoint is not for backing up tables. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> How can users create a new table and restore data from the > >>>>>>> specified > >>>>>>>>>>>>>>> savepoint? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This reminds me of savepoints in Flink. Still, savepoint is > >> not > >>>>>>> for backing > >>>>>>>>>>>>>> up tables so I guess we don't need to support "restoring data" > >>>>>>> from a > >>>>>>>>>>>>>> savepoint. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Shammon FY <[email protected]> 于2023年5月17日周三 10:32写道： > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks Zelin for initiating this discussion. I have some > >>>>>>> comments: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 1. Should we introduce an option for savepoint path which > >> may be > >>>>>>>>>>>>>> different > >>>>>>>>>>>>>>> from 'warehouse'? Then users can backup the data of > >> savepoint. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 2. Will the savepoint copy data files from snapshot or only > >> save > >>>>>>> meta > >>>>>>>>>>>>>>> files? The description in the PIP "After we introduce > >> savepoint, > >>>>>>> we > >>>>>>>>>>>>>> should > >>>>>>>>>>>>>>> also check if the data files are used by savepoints." looks > >> like > >>>>>>> we only > >>>>>>>>>>>>>>> save meta files for savepoint. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 3. How can users create a new table and restore data from the > >>>>>>> specified > >>>>>>>>>>>>>>> savepoint? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>> Shammon FY > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wed, May 17, 2023 at 10:19 AM Jingsong Li < > >>>>>>> [email protected]> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks Zelin for driving. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Some comments: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 1. I think it's possible to advance `Proposed Changes` to > >> the > >>>>>>> top, > >>>>>>>>>>>>>>>> Public API has no meaning if I don't know how to do it. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 2. Public API, Savepoint and SavepointManager are not Public > >>>>>>> API, only > >>>>>>>>>>>>>>>> Flink action or configuration option should be public API. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 3.Maybe we can have a separate chapter to describe > >>>>>>>>>>>>>>>> `savepoint.create-interval`, maybe 'Periodically > >> savepoint'? It > >>>>>>> is not > >>>>>>>>>>>>>>>> just an interval, because the true user case is savepoint > >> after > >>>>>>> 0:00. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 4.About 'Interaction with Snapshot', to be continued ... > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>> Jingsong > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Tue, May 16, 2023 at 7:07 PM yu zelin < > >> [email protected] > >>>>>>>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi, Paimon Devs, > >>>>>>>>>>>>>>>>> I’d like to start a discussion about PIP-4[1]. In this > >>>>>>> PIP, I > >>>>>>>>>>>>>> want > >>>>>>>>>>>>>>>> to talk about why we need savepoint, and some thoughts about > >>>>>>> managing > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>> using savepoint. Look forward to your question and > >> suggestions. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>> Yu Zelin > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/confluence/x/NxE0Dw > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > >> >
