Hi, Lari Thanks for driving the discussion, and I agree that the cherry-picking is the pain especially when we need to maintain old branches for a long time.
Frankly, my first impression is to target the bug fix to branch-3.0, but the features and improvements to the master branch will burden the contributors and committers more. They might merge the changes to the wrong branches for a time because they need time to build muscle memory. Of course, we can use CI to check the labels and the target branch. It will not be a blocker. I agree that the merge branch solution will resolve the ordering and coordination issues arising from the cherry-pick solutions. Coordination means how to decide a PR should be cherry-picked (Yunze pointed out to me). I have a few questions about the merge branch solution. - It looks like we will employ both merge branches and cherry-pick solutions finally after we have 4.0. Because at that time, the target branch for the BUG fix is branch-4.0, and we still have 18 month overlap. - For the existing cherry-picking solution, if there is a case that we can't cherry-pick it due to too many conflicts, we will usually create a separate PR for the release branch directly. How do we handle this case with the merge branch solution? If I understand correctly, we can also push separate PRs to the new branches and always apply the new branches when handling merge conflicts from this commit? - Is it possible to cherry-pick commits from the master to the LTS branch? The reason for asking this question is a PR might be recognized as an improvement, but someone found it should be contained in the LTS version. For example, https://github.com/apache/pulsar/pull/21739. Maybe there are other solutions to handle this case, e.g., push PR directly. Because we might get much more conflicts at that time. - Do we need to wait for the PRs that are targeted to branch-3.0 to be merged before cutting branch-4.0? Because if there are many comments on the existing PR, we don't want to ask the author to create a new one to continue the review with targeting branch-4.0. Usually, we will cut branches for preparing the release for at least 3 weeks. It sounds like a challenge because we will only allow regression fixes to branch-4.0 during that time. We need to find a solution for it. - Does the committer performing the branch merging need to resolve all the conflicts? I mean, if we have 20 commits need to merge, and maybe there is only one that is urgent to merge to the new branch for a patch release. With the cherry-pick solution, you can only cherry-pick that commit and create the patch release. I think we must merge all the commits for the merge branch solution. Maybe I'm wrong. I would support the merge branch solution and we also need documentation to clarify the items to note. If I understand correctly, we can also go back to the current solution if we find something is not working, right? Because the cherry-pick is very flexible even if the merges happen between branches. At least worth trying. Regards, Penghui On Wed, Mar 20, 2024 at 9:38 PM Yunze Xu <x...@apache.org> wrote: > > However, in async work, people should have more patience to read and > write. > > I mean, it would be better to have something like "TL; DR". Anyway, > I'd like to apply this change since the next feature release (3.3.0). > > Thanks, > Yunze > > On Tue, Mar 19, 2024 at 12:10 AM Lari Hotari <lhot...@apache.org> wrote: > > > > Thanks for the comments, Yunze. > > > > On 2024/03/18 05:48:39 Yunze Xu wrote: > > > I'm afraid many people don't have patience to read all the contents. > > > > I agree. However, in async work, people should have more patience to > read and write. Synchronous meetings aren't a good solution either. The > lack of patience could be caused by lack of interest. There's not a large > group of people in our community that are interested in improving the > maintenance strategy and also committed to invest their time and effort in > these activities. I hope more people sign up to this type of efforts and > show their interest and commitment in improving Apache Pulsar. > > > > > Here is my summary in short (please correct me if I'm wrong): > > > - For bug fixes, the target branch should be branch-3.0. Once the PR > > > is merged into branch-3.0, checkout the branch-3.x and run `git merge > > > branch-3.0` and resolve the conflicts > > > > I didn't describe the details of how this is handle. It is different in > practice. > > > > > - For features, the target branch should be branch-3.x > > > > New features would continue to go to master (or "main" if we decide to > rename it). Bugs would be fixed in the branch where the feature containing > the bug was introduced if it is missing from the LTS branch. > > > > > Since we introduced the LTS concept, I agree that we should make > > > branch-3.0 as the default branch. Cherry-picking is a disaster when > > > cherry-picks happen in the wrong order. > > > > Yes. > > > > -Lari > > > > On 2024/03/18 05:48:39 Yunze Xu wrote: > > > I'm afraid many people don't have patience to read all the contents. > > > Here is my summary in short (please correct me if I'm wrong): > > > - For bug fixes, the target branch should be branch-3.0. Once the PR > > > is merged into branch-3.0, checkout the branch-3.x and run `git merge > > > branch-3.0` and resolve the conflicts > > > - For features, the target branch should be branch-3.x > > > > > > Since we introduced the LTS concept, I agree that we should make > > > branch-3.0 as the default branch. Cherry-picking is a disaster when > > > cherry-picks happen in the wrong order. > > > > > > > > > Thanks, > > > Yunze > > > > > > On Tue, Mar 5, 2024 at 8:38 PM Lari Hotari <lhot...@apache.org> wrote: > > > > > > > > To enhance our maintenance processes, I've created a guide for > > > > configuring "git mergetool" to resolve merge conflicts: > > > > > > > > https://pulsar.apache.org/contribute/setup-mergetool/ > > > > > > > > For Apache Pulsar core developers, managing git merge conflict > > > > resolution is a necessary task. To streamline this process, it's > crucial > > > > to set up tools that aid in visualizing and resolving these > conflicts. > > > > > > > > I encourage you to follow the guide to set up a git mergetool. Your > > > > feedback is valuable, and you're welcome to contribute improvements > > > > directly to the website. You can do this by creating a PR by editing > > > > > https://github.com/apache/pulsar-site/edit/main/contribute/setup-mergetool.md > > > > directly in your browser. > > > > > > > > -Lari > > > > > > > > On 2024/03/01 14:01:55 Lari Hotari wrote: > > > > > Dear Pulsar Community, > > > > > > > > > > As we prepare for new releases in our maintenance branches, we > have once > > > > > again encountered issues with our cherry-picking process. Some of > our > > > > > maintenance branches are currently broken or were recently broken, > > > > > containing compilation errors or failing tests. Many have > encountered > > > > > these issues, as we have seen new PRs come in to address the > > > > > problems. The compilation problems are already being addressed by > > > > > Heesung (release manager for 3.0.3) and myself. We aim to resolve > these > > > > > issues as soon as possible. Please join #dev channel on Apache > Pulsar > > > > > Slack to collaborate in real time to help with this and get > updates. > > > > > > > > > > The cherry-picking process has always been problematic and lacks > clear > > > > > documentation in Apache Pulsar. This often leads to our maintenance > > > > > branches breaking, especially as we approach release dates and > begin > > > > > cherry-picking fixes. This recurring issue has been the subject of > > > > > multiple discussions over the years. The "feature freeze" in the > release > > > > > process does not mitigate the key problem with the cherry-picking > > > > > approach. > > > > > > > > > > Furthermore, the cherry-picking process is mostly based on tribal > > > > > knowledge and lacks clear documentation. I have previously > expressed my > > > > > concerns about this on the mailing list in this thread: > > > > > https://lists.apache.org/thread/69mwjso51kzkrv5xgdmw04d9wngbg8br > > > > > > > > > > Many problems with cherry-picking arise because cherry-picks occur > in > > > > > the wrong order, or dependent changes are not picked. Some > dependent > > > > > changes shouldn't be picked since when we have made bug fixes in > the > > > > > master branch, it can already contain changes for new features that > > > > > shouldn't be applied to maintenance branches. In those cases > > > > > a backport of the fix is needed and the original developer of the > > > > > PR might not be available to do this and there could be a > significant > > > > > delay for the release if delivering the backport takes time. > > > > > > > > > > When cherry-picking and backporting is delegated to other > developers, > > > > > in addition to delays, it can lead to coordination problems and > commits > > > > > being picked and applied in an order that results in even more > merge > > > > > conflicts. Thankfully, this isn't usually too painful, but it does > > > > > happen once in a while. > > > > > > > > > > A few days ago, I began working on improving the documentation of > the > > > > > current process. I have added a section where I share some > thoughts and > > > > > a tool to prevent future problems. You can find the document here: > > > > > > https://pulsar.apache.org/contribute/release-process/#cherry-picking-changes-scheduled-for-the-release > . > > > > > However, this does not fully describe the current process and will > only > > > > > help to some extent. > > > > > > > > > > The added section should help prevent cherry-picking in the wrong > order, > > > > > but it still has many gaps. Many developers do not have proper > merge > > > > > conflict resolution tools configured. Without proper 3-way diff > > > > > visualization and merge tools, it's very difficult to resolve many > of > > > > > the merge conflicts without making mistakes. This also requires a > deep > > > > > understanding of the module where the conflicts occur. > > > > > > > > > > After we have made the next set of maintenance releases, I plan to > > > > > propose an alternative to the cherry-picking process that will > address > > > > > the main issues that the Apache Pulsar project has been struggling > with > > > > > every time we do releases. > > > > > > > > > > The alternative would be to designate the LTS branch as the default > > > > > branch, make bug fixes primarily in the LTS branch, merge fixes to > newer > > > > > branches, and cherry-pick to possible older branches. This common > > > > > approach in many projects leverages what Git does well: handling > > > > > development across multiple branches. This solution ensures that > our LTS > > > > > branch is always immediately in a releasable state and the branch > will > > > > > also become the most stable version of Pulsar since bug fixes are > > > > > continuously evaluated and integrated into the LTS branch with our > CI > > > > > where bug fix PRs are targeted to the LTS branch. > > > > > Stability was the original goal of PIP-175 where the LTS concept > was > > > > > introduced to Pulsar. > > > > > > > > > > I hope that our community would be open to making changes to the > > > > > maintenance strategy to help resolve the pain that we have to deal > with > > > > > each time we make releases. Sometimes, this "cherry-picking vs. > merging > > > > > branches" discussion becomes a "tabs vs. spaces" type of pointless > > > > > discussion where personal preferences are emphasized. I hope that > we can > > > > > avoid that and admit the fact that releasing Apache Pulsar LTS > with this > > > > > cherry-picking process is a pain and we must fix it to make > progress as > > > > > a development community. > > > > > > > > > > -Lari > > > > > > > > >
