The feature `Replace Main With Branch` is used in duplicate data correction without modifying jobs. For example:
1. We can create branches with the same name for a series of paimon tables 2. Re-submit all streaming jobs to read and write these branches for tables 3. After the data in the branch is up to the main, we can stop all the jobs which read and write main branch 4. Replace main branch with the created branch, we don't need to do anything with the jobs read and write the specified branch We cannot `Merge Branch To Main` here because the correct jobs will still read and write the branches which will be completely independent of main. Best, Shammon FY On Thu, Sep 21, 2023 at 12:21 AM Jingsong Li <[email protected]> wrote: > Can you explain more about "Replace Main With Branch"? > > Does this need to be implemented? > > Best, > Jingsong > > On Tue, Sep 19, 2023 at 2:17 PM Shammon FY <[email protected]> wrote: > > > > Hi ConradJam, > > > > How to handle data conflicts between the main branch and branches is a > > complex problem. At present, we would like to replace data in main with > > branch directly. You can think that during merge and replace operations, > > the data after the specified tag in the main branch will be deleted and > > then the data after the tag in the branch will be used in the main. > > > > We can consider "merge" conflicting data in the future when we meet > these > > requirements. > > > > Best, > > Shammon FY > > > > On Tue, Sep 19, 2023 at 10:50 AM ConradJam <[email protected]> wrote: > > > > > +1 This feature looks a bit like Git’s branch management.If this is > really > > > the case, how do we solve the data conflict when merging branches? Do > we > > > need the user to specify that a certain branch data shall prevail? > > > > > > Shammon FY <[email protected]> 于2023年9月18日周一 20:06写道： > > > > > > > Hi Jingsong, > > > > > > > > I have updated the PIP-9 to explain that the main `Snapshot`, > `Schema` > > > and > > > > `Tag` will exist in the base directory by default, just as same as > the > > > > current directory structure. Thanks > > > > > > > > Best, > > > > Shammon FY > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:32 AM Shammon FY <[email protected]> > wrote: > > > > > > > > > Hi Jingsong, > > > > > > > > > > Thanks for your suggestion, it sounds good to me. Currently I only > > > > > mentioned it in the `Compatibility` section, I'll update the PIP to > > > > explain > > > > > this more clearly. > > > > > > > > > > Best, > > > > > Shammon FY > > > > > > > > > > On Wed, Sep 13, 2023 at 12:26 PM Jingsong Li < > [email protected]> > > > > > wrote: > > > > > > > > > >> Thanks Shammon for the proposal! > > > > >> > > > > >> It looks very good! > > > > >> > > > > >> I don't get the main branch file. > > > > >> > > > > >> Can we keep the main branch as it is? Just put snapshot/ tag/ > schema/ > > > > >> in the table root directory. > > > > >> > > > > >> Best, > > > > >> Jingsong > > > > >> > > > > >> On Tue, Sep 12, 2023 at 3:55 PM Shammon FY <[email protected]> > wrote: > > > > >> > > > > > >> > Hi devs, > > > > >> > > > > > >> > I would like to start a discussion about PIP-9: Support Branch > [1]. > > > > >> Branch > > > > >> > in Paimon will help us deal with data correction without > copying all > > > > >> data > > > > >> > from original tables, and it can also enhance Tag for Paimon > like > > > > >> > traditional Hive partition tables, providing data correction > > > > >> capabilities > > > > >> > on the basis of Tag. > > > > >> > > > > > >> > Looking forward to your feedback, thanks! > > > > >> > > > > > >> > > > > > >> > [1] > > > > >> > > > > > >> > > > > > > > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-9%3A+Support+Branch > > > > >> > > > > > >> > Best, > > > > >> > Shammon FY > > > > >> > > > > > > > > > > > > > > > > > > -- > > > Best > > > > > > ConradJam > > > >
