I do not think estimations should be the only driver to make a decision, especially when the current proposal is conceptually incompatible with the multi repo approach that is taken elsewhere in the project. Given my knowledge of the CI it is nearly impossible for me to give a fair estimate of how much it might take to achieve step 2) of my previous e-mail . It might take a while or it might be pretty easy after all, I don't really know, but I think it will be a good idea if some of the experts on CI in the team (the ones that set up the pipeline, which was a huge achievement) give an estimate, not me. Estimating how much it takes to merge two existing repos (without altering CI) is easier, but it does not mean we are doing the right thing. My main concern is that it will be very difficult for me to explain to someone that arrives new to the team, that having experts on CI on the team, we decided to merge two repos (once merged, it would be rather difficult to unmerge) rather than fix the CI, because of expediency.
On Wed, Mar 13, 2024 at 12:30 PM Alex Porcelli <porce...@apache.org> wrote: > Francisco, > > Please take the time to make the more in depth analysis needed and provide > a more detailed plan… so we - as community- can evaluate the size of the > effort. In the conceptual level you shared it’s near impossible to estimate > the size of the effort and compare with the current proposal. > > On Wed, Mar 13, 2024 at 7:23 AM Francisco Javier Tirado Sarti < > ftira...@redhat.com> wrote: > > > I think I already did a high level proposal. > > 1) Remove all dependencies from tooling to images, so images depend on > > tooling but tooling does not depend on images. > > 2) Then change CI to deal with tooling repo before dealing with images > > repo. > > I understand that CI details are tricky and since I'm not familiar with > CI > > in any way, I barely can make a low level design, but we do not need to > fix > > everything, just achieve 2), a change of compilation order. > > > > > > On Wed, Mar 13, 2024 at 12:17 PM Alex Porcelli <a...@porcelli.me> wrote: > > > > > Francisco and Grabriele, > > > > > > You may not like or understand why the current state of the CI is like > > > that… actually has been in Red Hat and has been replicated in Apache as > > > used to be…. > > > > > > But the fact is that this is the current reality. > > > > > > If you disagree with the current plan, please provide a detailed > > > alternative so we, as community, can better evaluate the pros and cons > of > > > each proposal. > > > > > > > > > I think it’s also fair to say that, post 10 release we need to have a > > much > > > in depth discussion about how our codebase is organized, it’s clear > that > > > it’s not working. > > > > > > > > > On Wed, Mar 13, 2024 at 6:41 AM Gabriele Cardosi < > > > gabriele.card...@gmail.com> > > > wrote: > > > > > > > As Francisco said, > > > > I also have the impression that the "images" (if we are talking of > > docker > > > > images) should be the very last one to be built, in a standalone > repo. > > > > That way, they may "combine" artifacts that are built in different > > repos, > > > > regardless of the order in which those are built. > > > > Moving them out of all the repos (kogito-apps/kie-tools) maybe could > > > > simplify the situation a bit. > > > > (I also think there are some statements of undisputable needs while > > they > > > > are, actually, just technical choices. > > > > Anyway, this latter point is for longer, following, discussion.) > > > > > > > > > > > > > > > > > > > > Il giorno mer 13 mar 2024 alle ore 11:23 Francisco Javier Tirado > Sarti > > < > > > > ftira...@redhat.com> ha scritto: > > > > > > > > > Alex, > > > > > There are two assumptions that deserve further discussion: > > > > > 1) That tool has to be the last to build. why? it does not have > more > > > > sense > > > > > to build final images after everything else has been built?- > > > > > 2) That the impact (in terms of effort now) on fixing CI is bigger > > than > > > > the > > > > > impact (long term consequences) of consolidating two unrelated > piece > > of > > > > > software within the same repository. > > > > > > > > > > > > > > > On Wed, Mar 13, 2024 at 11:15 AM Alex Porcelli <a...@porcelli.me> > > > wrote: > > > > > > > > > > > Francisco, > > > > > > > > > > > > This was discussed as an alternative solution, however it has > major > > > > > impact > > > > > > on CI and there’s also the fact Tool has been always the last to > > > build > > > > > and > > > > > > has no Snapshot published (actually in JavaScript world there is > no > > > > > > snapshot concept). > > > > > > > > > > > > So, based on our evaluation… the proposal here is the least > > > disruptive > > > > > and > > > > > > will take less time to unblock the release. > > > > > > > > > > > > Regards, > > > > > > _____________ > > > > > > Alex Porcelli > > > > > > http://porcelli.me > > > > > > > > > > > > > > > > > > On Wed, Mar 13, 2024 at 6:09 AM Francisco Javier Tirado Sarti < > > > > > > ftira...@redhat.com> wrote: > > > > > > > > > > > > > After kie-tools, sorry. I think we are not embracing the fact > > that > > > > > > > kogito-images depend on kie-tools, because we want those images > > to > > > > > > include > > > > > > > tools. > > > > > > > > > > > > > > On Wed, Mar 13, 2024 at 11:08 AM Francisco Javier Tirado Sarti > < > > > > > > > ftira...@redhat.com> wrote: > > > > > > > > > > > > > > > Hi Tiago, > > > > > > > > It can be an alternative solution to move kn-plugin-workflow > to > > > > > > > > kogito-images (so there is not longer dependency from tools > to > > > > > images) > > > > > > > and > > > > > > > > then build kogito-images after kogito-tools? > > > > > > > > > > > > > > > > On Wed, Mar 13, 2024 at 11:01 AM Enrique Gonzalez Martinez < > > > > > > > > egonza...@apache.org> wrote: > > > > > > > > > > > > > > > >> +1 to unblock release > > > > > > > >> > > > > > > > >> El mié, 13 mar 2024, 10:48, Pere Fernandez (apache) < > > > > > > > pefer...@apache.org> > > > > > > > >> escribió: > > > > > > > >> > > > > > > > >> > I say +1 in order to move forward with the 10. > > > > > > > >> > > > > > > > > >> > On Tue, 12 Mar 2024 at 21:45, Alex Porcelli < > > a...@porcelli.me > > > > > > > > > > wrote: > > > > > > > >> > > > > > > > > >> > > +1 > > > > > > > >> > > > > > > > > > >> > > I spent the last day or so working closely with Tiago, > > > > exploring > > > > > > > >> > different > > > > > > > >> > > options and getting deeper on the impact and evaluating > > the > > > > > > overall > > > > > > > >> > release > > > > > > > >> > > procedure steps required. I agree with the proposal as > the > > > > most > > > > > > > >> > > viable option for unblocking the 10 release in the > > > reasonable > > > > > time > > > > > > > >> frame. > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > On Tue, Mar 12, 2024 at 3:45 PM Tiago Bento < > > > > > > tiagobe...@apache.org> > > > > > > > >> > wrote: > > > > > > > >> > > > > > > > > > >> > > > Hi everyone, > > > > > > > >> > > > > > > > > > > >> > > > Unfortunately, I can't do a tl;dr this time, as this > > > matter > > > > > > > >> requires a > > > > > > > >> > > > lot of context. > > > > > > > >> > > > > > > > > > > >> > > > This email will take you < 20 minutes to read, > according > > > to > > > > > > > >> > > > https://thereadtime.com/. > > > > > > > >> > > > > > > > > > > >> > > > As you may have followed on a separate thread > > > > > > > >> > > > ( > > > > > > https://lists.apache.org/thread/nknm6j641qk2c7cl621tsy3fy98tsc69 > > > > > > > ), > > > > > > > >> > > > many of us were working towards removing a circular > > > > dependency > > > > > > > >> > > > currently present between `kogito-apps` and > `kie-tools`. > > > As > > > > we > > > > > > > >> > > > progressed towards a solution, we kept finding the > > > circular > > > > > > > >> dependency > > > > > > > >> > > > pop up somewhere else. I'll do a breakdown of the > things > > > we > > > > > did, > > > > > > > and > > > > > > > >> > > > the results we had. > > > > > > > >> > > > > > > > > > > >> > > > Right now, even though we started the effort to move > the > > > > > Quarkus > > > > > > > Dev > > > > > > > >> > > > UI modules to `kie-tools`, we haven't been able to do > it > > > > yet, > > > > > as > > > > > > > >> we've > > > > > > > >> > > > been busy upgrading KIE Tools to Java 17, Maven 3.9.6, > > and > > > > > > Quarkus > > > > > > > >> > > > 3.2.9, compatible with Kogito Runtimes > > > > 999-20240218-SNAPSHOT. > > > > > > This > > > > > > > >> > > > effort was concluded this Monday, with > > > > > > > >> > > > > https://github.com/apache/incubator-kie-tools/pull/2136 > > . > > > > > > > >> > > > > > > > > > > >> > > > The current scenario we have is: > > > > > > > >> > > > > > > > > > > >> > > > 01. incubator-kie-kogito-runtimes > > > > > > > >> > > > |==> 02. incubator-kie-kogito-apps > > > > > > > >> > > > C | 03. incubator-kie-kogito-examples > > > > > > > >> > > > Y | 04. incubator-kie-kogito-images > > > > > > > >> > > > C | 05. > > > incubator-kie-kogito-serverless-operator > > > > > > > >> > > > L | ========================== > > > > > > > >> > > > E | 06. > > > > incubator-kie-sandbox-quarkus-accelerator > > > > > > > >> > > > |==> 07. incubator-kie-tools > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > * As `kie-tools`/extended-services depends on > > > > > > > >> > > > `kogito-apps`/jitexecutor; > > > > > > > >> > > > * and > > > `kogito-apps`/{sonataflow,bpmn}-quarkus-devui > > > > > > depend > > > > > > > >> on > > > > > > > >> > > > `kie-tools`/{many packages} > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > After moving the Quarkus Dev UIs to `kie-tools`, we > > > would've > > > > > > had: > > > > > > > >> > > > > > > > > > > >> > > > 01. incubator-kie-kogito-runtimes > > > > > > > >> > > > 02. incubator-kie-kogito-apps > > > > > > > >> > > > 03. incubator-kie-kogito-examples > > > > > > > >> > > > C |==> 04. incubator-kie-kogito-images > > > > > > > >> > > > Y | 05. > > > incubator-kie-kogito-serverless-operator > > > > > > > >> > > > C | ===================== > > > > > > > >> > > > L | 06. > > > > incubator-kie-sandbox-quarkus-accelerator > > > > > > > >> > > > E |==> 07. incubator-kie-tools > > > > > > > >> > > > > > > > > > > >> > > > * As `kie-tools`/kn-plugin-workflow depends on > > > > > > > >> > > > `kogito-images`/kogito-swf-devmode; > > > > > > > >> > > > * and `kogito-images`/kogito-swf-devmode > depends > > > on > > > > > > > >> > > > `kie-tools`/sonataflow-quarkus-devui > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > After moving the `kogito-swf-devmode` image to > > > `kie-tools`, > > > > we > > > > > > > >> would've > > > > > > > >> > > > had: > > > > > > > >> > > > > > > > > > > >> > > > 01. incubator-kie-kogito-runtimes > > > > > > > >> > > > 02. incubator-kie-kogito-apps > > > > > > > >> > > > 03. incubator-kie-kogito-examples > > > > > > > >> > > > 04. incubator-kie-kogito-images > > > > > > > >> > > > C |==> 05. > > incubator-kie-kogito-serverless-operator > > > > > > > >> > > > Y | ===================== > > > > > > > >> > > > C | 06. > > > > incubator-kie-sandbox-quarkus-accelerator > > > > > > > >> > > > L |==> 07. incubator-kie-tools > > > > > > > >> > > > E > > > > > > > >> > > > > > > > > > > >> > > > * As `kie-tools`/kn-plugin-workflow depends on > > > > > > > >> > > > `kogito-serverless-operator`; > > > > > > > >> > > > * and `kogito-serverless-operator` depends on > > > > > > > >> > > > `kie-tools`/kogito-swf-devmode > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > Clearly, we have a much bigger problem than a simple > > > > circular > > > > > > > >> > dependency. > > > > > > > >> > > > > > > > > > > >> > > > After multiple conversations with a lot of people, > it's > > > been > > > > > > > really > > > > > > > >> > > > hard coming up with a simple solution that makes it > > > possible > > > > > to > > > > > > > >> build > > > > > > > >> > > > Apache KIE in one shot, while preserving the way > > everyone > > > is > > > > > > used > > > > > > > to > > > > > > > >> > > > contributing to the multiple repositories we have. > More > > > than > > > > > > that, > > > > > > > >> > > > while making this assessment, I found more problems > > that, > > > in > > > > > my > > > > > > > >> > > > perspective, block Apache KIE 10. > > > > > > > >> > > > > > > > > > > >> > > > In light of that difficulty, I'm coming forward with > my > > > > > proposal > > > > > > > for > > > > > > > >> > > > the Apache KIE release process, so we can use Apache's > > > > > > mechanisms > > > > > > > to > > > > > > > >> > > > have a slower-paced, in-depth debate about this really > > > > > > complicated > > > > > > > >> > > > matter. > > > > > > > >> > > > > > > > > > > >> > > > I'll lay out my entire perspective about the current > > > > situation > > > > > > of > > > > > > > >> our > > > > > > > >> > > > codebase, as well as problems I can currently see. > I'll > > > > start > > > > > > with > > > > > > > >> an > > > > > > > >> > > > analysis of the repositories and their purposes, point > > out > > > > > some > > > > > > > >> > > > problems that I believe are blocking our 10 release, > > > explain > > > > > my > > > > > > > >> > > > proposal and discuss some consequences to what I'm > > > > proposing. > > > > > > > >> > > > > > > > > > > >> > > > Let's begin. > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > # THE APACHE KIE REPOS > > > > > > > >> > > > > > > > > > > >> > > > A. DROOLS OPTAPLANNER, & KOGITO (count: 11) > > > > > > > >> > > > - incubator-kie-kogito-pipelines @ `main` > > > > > > > >> > > > - incubator-kie-drools @ `main` > > > > > > > >> > > > - incubator-kie-optaplanner @ `main` > > > > > > > >> > > > - incubator-kie-optaplanner-quickstarts @ `main` > > > > > > > >> > > > - incubator-kie-kogito-runtimes @ `main` > > > > > > > >> > > > - incubator-kie-kogito-apps @ `main` > > > > > > > >> > > > - incubator-kie-kogito-examples @ `main` > > > > > > > >> > > > - incubator-kie-kogito-images @ `main` > > > > > > > >> > > > - incubator-kie-kogito-serverless-operator @ `main` > > > > > > > >> > > > - incubator-kie-kogito-docs @ `main` > > > > > > > >> > > > - incubator-kie-docs @ `main-kogito` > > > > > > > >> > > > > > > > > > > >> > > > B. TOOLS (count: 2) > > > > > > > >> > > > - incubator-kie-sandbox-quarkus-accelerator @ `0.0.0` > > > > > > > >> > > > - incubator-kie-tools @ `main` > > > > > > > >> > > > > > > > > > > >> > > > C. BENCHMARKS (count: 2) > > > > > > > >> > > > - incubator-kie-kogito-benchmarks @ `main` > > > > > > > >> > > > - incubator-kie-benchmarks @ `main` > > > > > > > >> > > > > > > > > > > >> > > > D. ARCHIVED (count: 1) > > > > > > > >> > > > - incubator-kie-kogito-operator > > > > > > > >> > > > > > > > > > > >> > > > E. "NON-CODE" (count: 5) > > > > > > > >> > > > - incubator-kie-issues @ `main` > > > > > > > >> > > > (Issues only, README should be updated @ `main`. > > Same > > > > for > > > > > > > GitHub > > > > > > > >> > > > Actions workflows.) > > > > > > > >> > > > - incubator-kie-kogito-website @ `main` > > > > > > > >> > > > (The Kogito website. Develop & deploy at the > `main` > > > > > branch.) > > > > > > > >> > > > - incubator-kie-website @ `main` > > > > > > > >> > > > (The KIE website. Develop @ `main`. Push @ > `deploy` > > to > > > > > > update > > > > > > > >> the > > > > > > > >> > > > website.) > > > > > > > >> > > > - incubator-kie-kogito-online @ `gh-pages` > > > > > > > >> > > > (GitHub pages used to host sandbox.kie.org and > KIE > > > > Tools' > > > > > > > >> Chrome > > > > > > > >> > > > Extension assets.) > > > > > > > >> > > > - incubator-kie-kogito-online-staging @ `main` > > > > > > > >> > > > (Same as above, but for manual sanity checks > during > > > the > > > > > > > staging > > > > > > > >> > > > phase of a release.) > > > > > > > >> > > > > > > > > > > >> > > > TOTAL (count: 21) > > > > > > > >> > > > > > > > > > > >> > > > I grouped the repositories by category, and listed > them > > > in a > > > > > > > >> > > > topological order. Keep in mind that when flattening > > out a > > > > > tree, > > > > > > > >> there > > > > > > > >> > > > are multiple possibilities. For example, OptaPlanner > > > > could've > > > > > > been > > > > > > > >> > > > placed in any position after Drools. > > > > > > > >> > > > > > > > > > > >> > > > Category A repos are what I've been referring to as > > > `drools` > > > > > and > > > > > > > >> > > > `kogito-*` stream. Of course OptaPlanner is inside > that > > > > > stream, > > > > > > as > > > > > > > >> the > > > > > > > >> > > > way these repositories reference each other are > through > > > > Maven > > > > > > > >> > > > SNAPSHOTs. More specifically, the 999-SNAPSHOT > version. > > > This > > > > > > > >> mechanism > > > > > > > >> > > > is well-known to the team, and although flawed for > > > intra-day > > > > > > > builds > > > > > > > >> > > > and disruptive for people in many different time > zones, > > it > > > > is > > > > > > > >> already > > > > > > > >> > > > very comfortable for everyone to work with, I assume. > > > > > > > >> > > > > > > > > > > >> > > > Contributions made to Category A have some dedicated > > > > > pipelines, > > > > > > > >> which > > > > > > > >> > > > are, at least to some extent, able to build cross-repo > > PRs > > > > > > > together > > > > > > > >> > > > and verify that the codebase will continue working as > > > > expected > > > > > > > after > > > > > > > >> > > > they're all merged. From what I could gather, there > are > > > some > > > > > > > >> > > > "sub-streams" currently configured for cross-repo PRs. > > > > > > > >> > > > > > > > > > > >> > > > - kogito-pipelines > > > > > > > >> > > > - drools, kogito-runtimes, kogito-apps, and > > > kogito-examples > > > > > > > >> > > > - optaplanner, and optaplanner-quickstarts > > > > > > > >> > > > - kogito-images, and kogito-serverless-operator > > > > > > > >> > > > - kogito-docs > > > > > > > >> > > > - kie-docs > > > > > > > >> > > > > > > > > > > >> > > > This means that sending cross-repo PRs to any > > combination > > > of > > > > > > repos > > > > > > > >> > > > that are not part of the same "sub-stream" cannot be > > > > verified > > > > > > > before > > > > > > > >> > > > merging, making our contribution model dependent on > > > > individual > > > > > > > >> > > > contributors building stuff on their machines to > verify > > > that > > > > > it > > > > > > > >> works. > > > > > > > >> > > > > > > > > > > >> > > > I based this analysis on > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-kogito-pipelines/blob/main/.ci/project-dependencies.yaml > > > > > > > >> > > > , > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-optaplanner/blob/main/.ci/buildchain-project-dependencies.yaml > > > > > > > >> > > > , > > > > > > > >> > > > and > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-kogito-pipelines/blob/main/.ci/jenkins/config/branch.yaml > > > > > > > >> > > > . > > > > > > > >> > > > Note that I'm not that familiar with these pipelines, > so > > > > > please > > > > > > > >> > > > someone correct me if I'm wrong. > > > > > > > >> > > > > > > > > > > >> > > > Category B repos are what I've been referring to as > > > > > `kie-tools` > > > > > > > >> > > > stream. The first repo there is a template repository > > that > > > > is > > > > > > used > > > > > > > >> by > > > > > > > >> > > > people starting projects from scratch on KIE Sandbox, > > > > similar > > > > > > to a > > > > > > > >> > > > Maven archetype, if you will. The other one is the KIE > > > Tools > > > > > > > >> monorepo, > > > > > > > >> > > > a polyglot monorepo with `pnpm` as its build system. > > > > > Currently, > > > > > > > KIE > > > > > > > >> > > > Tools hosts Java libraries and apps, TypeScript > > libraries > > > > and > > > > > > > apps, > > > > > > > >> Go > > > > > > > >> > > > apps, Docker images, and Helm charts. The `kie-tools` > > > > monorepo > > > > > > is > > > > > > > >> > > > configured to work with sparse checkouts and can do > > > partial > > > > > > > builds. > > > > > > > >> > > > Category B repos refer to Category A repos through > > > > timestamped > > > > > > > >> > > > SNAPSHOTs. This is a new mechanism we recently > > introduced > > > > that > > > > > > > will > > > > > > > >> > > > build and publish immutable, persistent artifacts > under > > a > > > > > > version > > > > > > > >> > > > following the 999-YYYYMMDD-SNAPSHOT format, published > > > weekly > > > > > > every > > > > > > > >> > > > Sunday night. Timestamped SNAPSHOTs are an evolution > to > > > the > > > > > > Kogito > > > > > > > >> > > > releases, as we're now targeting one release for all > of > > > > Apache > > > > > > > KIE, > > > > > > > >> so > > > > > > > >> > > > we can't have Kogito releases anymore. > > > > > > > >> > > > > > > > > > > >> > > > An important note here is that Category B repositories > > > have > > > > > been > > > > > > > >> > > > historically kept out of any automations we used to > > have, > > > > way > > > > > > back > > > > > > > >> > > > when Kogito started and we had the Business Central > > > (a.k.a. > > > > > v7) > > > > > > > >> stream > > > > > > > >> > > > still going on. For this reason, Category B projects > > have > > > > > > > developed > > > > > > > >> > > > their own automations, based on GitHub Actions. > > Category B > > > > > repos > > > > > > > >> have > > > > > > > >> > > > always depended on Category A repos using fixed > > versions. > > > If > > > > > > > >> Category > > > > > > > >> > > > B repos have had adopted mutable SNAPSHOTs, breaking > > > changes > > > > > on > > > > > > > >> > > > Category A repositories would've had the potential to > > > break > > > > > > > >> Category B > > > > > > > >> > > > silently, leaving Category B with a broken development > > > > stream, > > > > > > and > > > > > > > >> > > > introducing unpleasant surprises for maintainers of > > > > Category B > > > > > > > >> repos, > > > > > > > >> > > > as historically Category A contributors were not > > familiar > > > > with > > > > > > > >> > > > Category B repos. > > > > > > > >> > > > > > > > > > > >> > > > Contributions made to Category B repos go through a > > GitHub > > > > > > Actions > > > > > > > >> > > > workflow that builds the relevant part of the > > `kie-tools` > > > > > > monorepo > > > > > > > >> for > > > > > > > >> > > > the changes introduced. Changes made to the pipeline > > > itself > > > > > are > > > > > > > also > > > > > > > >> > > > picked up as part of PRs, allowing us to do things > like > > > > > > atomically > > > > > > > >> > > > bumping the Node.js version, for example. More > > > importantly, > > > > it > > > > > > > >> allows > > > > > > > >> > > > us to upgrade the repository to a new timestamped > > SNAPSHOT > > > > > > > together > > > > > > > >> > > > with the changes necessary to make it stay green. > > > > > > > >> > > > > > > > > > > >> > > > This setup, however, makes it impossible to have > > > cross-repo > > > > > PRs > > > > > > > >> > > > involving Category A and Category B simultaneously, > with > > > the > > > > > > > current > > > > > > > >> > > > automations we have. > > > > > > > >> > > > > > > > > > > >> > > > Category C repos are kind of floating around, and I'm > > not > > > > sure > > > > > > if > > > > > > > >> > > > there's much activity going on there. Regardless, as > > > they're > > > > > > part > > > > > > > of > > > > > > > >> > > > Apache KIE, they will be part of our release, so I > > listed > > > > them > > > > > > for > > > > > > > >> us > > > > > > > >> > > > to take them into consideration too. > > > > > > > >> > > > > > > > > > > >> > > > Category D is self explanatory. There's only one repo > > that > > > > has > > > > > > > >> already > > > > > > > >> > > > been marked for being archived. > > > > > > > >> > > > > > > > > > > >> > > > Category E are repos that do not host code directly, > and > > > are > > > > > > > either > > > > > > > >> > > > organizational entities, or host websites, that > > currently > > > > are > > > > > > not > > > > > > > >> part > > > > > > > >> > > > of any pipelines we have. > > > > > > > >> > > > > > > > > > > >> > > > This lack of unification between Category A and > > Category B > > > > is, > > > > > > > IMHO, > > > > > > > >> > > > what allowed us to introduce the infamous circular > > > > dependency > > > > > > > >> between > > > > > > > >> > > > `kie-tools` and `kogito-apps`, which we now can > describe > > > as > > > > a > > > > > > > >> circular > > > > > > > >> > > > dependency between Category A and Category B. The way > I > > > see > > > > > it, > > > > > > if > > > > > > > >> we > > > > > > > >> > > > had a single pipeline, building everything from > `drools` > > > to > > > > > > > >> > > > `kie-tools`, such flaws would've never been > introduced, > > > and > > > > we > > > > > > > >> > > > wouldn't be having this huge problem in our hands > right > > > now. > > > > > > > >> > > > > > > > > > > >> > > > My proposal for the Apache KIE release process sees > this > > > > lack > > > > > of > > > > > > > >> > > > unification as a central problem, not only for this > > > release > > > > in > > > > > > > >> > > > particular, but for the community as a whole. It is my > > > > belief > > > > > > that > > > > > > > >> we > > > > > > > >> > > > are all under the same roof, and that no contribution > > > should > > > > > be > > > > > > > >> > > > allowed to break any part of our codebase. With the > > > > increasing > > > > > > > >> volume > > > > > > > >> > > > of code, and hopefully number of contributors too, we > > > cannot > > > > > > keep > > > > > > > >> > > > counting on "common sense" to avoid breaking things. > > We're > > > > all > > > > > > > >> humans > > > > > > > >> > > > after all, and it is our job to have mechanisms in > place > > > to > > > > > > > prevent > > > > > > > >> us > > > > > > > >> > > > from unwillingly making mistakes. Especially when > these > > > > > mistakes > > > > > > > >> > > > impact on parts of the codebase that we, individually, > > > > > probably > > > > > > > >> can't > > > > > > > >> > > > fix. > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > # THE PROBLEMS WE HAVE RIGHT NOW > > > > > > > >> > > > > > > > > > > >> > > > P1. Quarkus Dev UIs @ `kogito-apps` depending on > > > kiegroup's > > > > > KIE > > > > > > > >> Tools > > > > > > > >> > > > `0.32.0`. > > > > > > > >> > > > See: > > > > > > > >> > > > - > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/search?q=repo%3Akiegroup%2Fkogito-apps+path%3Apackage.json+kie-tools&type=code > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > P2. PR open for Kogito SWF images @ `kogito-images` > > > > depending > > > > > on > > > > > > > >> > > > kiegroup's KIE Tools `0.32.0`. > > > > > > > >> > > > See: > > > > > > > >> > > > - > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-tools/tree/main/packages/sonataflow-deployment-webapp > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > P3. DashBuilder @ `kie-tools` depending on kiegroup's > > > > `lienzo` > > > > > > and > > > > > > > >> > > > `kie-soup` artifacts at version `7.59.0.Final`. > > > > > > > >> > > > See: > > > > > > > >> > > > - > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-tools/blob/main/packages/dashbuilder/pom.xml#L64 > > > > > > > >> > > > - > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/search?q=repo%3Aapache%2Fincubator-kie-tools+path%3Apackages%2Fdashbuilder+%24%7Bversion.org.kie%7D&type=code > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > P4. Multiple packages @ `kogito-apps` depending on > > > > kiegroup's > > > > > > > >> > > > Explainability `1.22.1.Final`. > > > > > > > >> > > > * This module was removed from the KIE codebase here: > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-kogito-apps/commit/bbb22c06d37e77b97aae6496d74abe43a8cfc965 > > > > > > > >> > > > and now lives on > > > > > > > >> > > > > > > > > > > > > https://github.com/trustyai-explainability/trustyai-explainability > > > , > > > > > > > >> > > > under a different GAV. > > > > > > > >> > > > * This new repo depends on Kogito and OptaPlanner, > > > pointing > > > > to > > > > > > > older > > > > > > > >> > > > versions. > > > > > > > >> > > > See: > > > > > > > >> > > > - > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/search?q=repo%3Aapache%2Fincubator-kie-kogito-apps+%3Eexplainability-core%3C&type=code > > > > > > > >> > > > - > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/trustyai-explainability/trustyai-explainability/blob/main/pom.xml#L52-L53 > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > P5. `incubator-kie-sandbox-quarkus-accelerator` > > depending > > > on > > > > > > > Kogito > > > > > > > >> > > > `1.32.0.Final` and Quarkus `2.15.3.Final`. > > > > > > > >> > > > See: > > > > > > > >> > > > - > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-sandbox-quarkus-accelerator/blob/0.0.0/pom.xml#L32-L38 > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > P6. Category C repos are out of date and not part of > the > > > > > > Category > > > > > > > A > > > > > > > >> > > > CI/Release pipelines. > > > > > > > >> > > > * incubator-kie-kogito-benchmarks: (Current version is > > > > > > > >> `2.0-SNAPSHOT`, > > > > > > > >> > > > depending on Kogito without a specific version, only > by > > > > using > > > > > > > >> > > > `http://localhost:8080`) > > > > > > > >> > > > * incubator-kie-benchmarks: (Current version is > > > > > `1.0-SNAPSHOT`, > > > > > > > >> > > > pointing to Drools 999-SNAPSHOT and OptaPlanner > > > > > > `8.45.0-SNAPSHOT`) > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > P7. `kie-tools`/packages/kn-plugin-workflow has its > E2E > > > > > disabled > > > > > > > >> after > > > > > > > >> > > > upgrading to 999-20240218-SNAPSHOT. > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > In my perspective, P1 and P2 have the same solution, > as > > > they > > > > > > both > > > > > > > >> > > > suffer from the circular dependency between Category A > > and > > > > > > > Category > > > > > > > >> B. > > > > > > > >> > > > As Category A and Category B are both streams that > have > > > been > > > > > > > really > > > > > > > >> > > > active, I see this as a blocker, as there are > > > contributions > > > > > that > > > > > > > >> > > > cannot be done, given that Category A depends on > > Category > > > B > > > > > > with a > > > > > > > >> > > > dephasing of 1 release. > > > > > > > >> > > > > > > > > > > >> > > > P3 and P4, although not ideal, can be understood as > > > > technical > > > > > > > debt. > > > > > > > >> > > > Depending on unmaintained projects is something we'll > > > always > > > > > be > > > > > > > >> > > > susceptible to, given time. > > > > > > > >> > > > > > > > > > > >> > > > P5 and P6 are easily fixable, as it's just a matter of > > > > making > > > > > > them > > > > > > > >> > > > part of the play. > > > > > > > >> > > > > > > > > > > >> > > > P7 is an isolated problem that won't impact the > > structure > > > or > > > > > > > >> anything > > > > > > > >> > > > that we're talking about here, but it is a regression > we > > > > > > > introduced > > > > > > > >> > > > recently. > > > > > > > >> > > > > > > > > > > >> > > > Assuming P3 and P4 can be ignored for Apache KIE 10, > and > > > > that > > > > > > P5, > > > > > > > >> P6, > > > > > > > >> > > > and P7 have easy fixes, the only problems left to > > discuss > > > > are > > > > > P1 > > > > > > > and > > > > > > > >> > > > P2, which can't be done without a proper proposal. > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > # THE PROPOSAL > > > > > > > >> > > > > > > > > > > >> > > > I'll try to be very meticulous here, since from my > > > > experience, > > > > > > any > > > > > > > >> > > > little miscalculation can lead to our release not > > working > > > > out > > > > > in > > > > > > > the > > > > > > > >> > > > end. To try and avoid that as much as possible, and > make > > > > > > > everything > > > > > > > >> we > > > > > > > >> > > > can to have a successful Apache KIE 10 release, bear > > with > > > > me. > > > > > > I'll > > > > > > > >> lay > > > > > > > >> > > > out a timeline of events that need to happen in order > > for > > > > our > > > > > > > >> release > > > > > > > >> > > > to be published, with all artifacts ending up in the > > right > > > > > > places, > > > > > > > >> but > > > > > > > >> > > > first, we need to solve problems P1 and P2. > > > > > > > >> > > > > > > > > > > >> > > > As you saw at the beginning of this email, all the > > > attempts > > > > we > > > > > > > made > > > > > > > >> > > > left us with the circular dependency showing up at a > > > > different > > > > > > > >> place, > > > > > > > >> > > > but something all these places have in common is that > > > > they're > > > > > > all > > > > > > > >> > > > after kogito-apps, and before to Category B. > > > > > > > >> > > > > > > > > > > >> > > > The first part of my proposal is the following: > > > > > > > >> > > > > > > > > > > >> > > > S1. We keep the original plan of moving the Quarkus > Dev > > > UIs > > > > > from > > > > > > > >> > > > `kogito-apps` to `kie-tools`, together with Management > > and > > > > > Task > > > > > > > >> > > > consoles from `kogito-images` to `kie-tools`. > > > > > > > >> > > > S2. We move the `kogito-swf-devmode` and > > > > `kogito-swf-builder` > > > > > > > images > > > > > > > >> > > > from `kogito-images` to `kie-tools` too. > > > > > > > >> > > > S3. We move the entire `kogito-serverless-operator` > repo > > > > > inside > > > > > > a > > > > > > > >> new > > > > > > > >> > > > package on `kie-tools`, keeping Git history. > > > > > > > >> > > > > > > > > > > >> > > > Solutions S1, S2, and S3 together solve problems P1 > and > > > P2. > > > > Of > > > > > > > >> course > > > > > > > >> > > > the rest of > > > > > > > >> https://github.com/apache/incubator-kie-issues/issues/967 > > > > > > > >> > > > would still be done too. > > > > > > > >> > > > > > > > > > > >> > > > This doesn't come without consequences, of course, as > > the > > > > > > > >> > > > `kogito-swf-devmode` and `kogito-swf-builder` images, > > and > > > > the > > > > > > > >> > > > `kogito-serverless-operator` would be moving from > > > Category A > > > > > to > > > > > > > >> > > > Category B. This move would make them have to > reference > > > > > > Category A > > > > > > > >> > > > repos through timestamped SNAPSHOTs. Since > > `kogito-images` > > > > and > > > > > > > >> > > > `kogito-serverless-operator` are already their own > > > > > "sub-stream" > > > > > > > >> inside > > > > > > > >> > > > Category A, though, contributions made in a cross-repo > > > > fashion > > > > > > to > > > > > > > >> this > > > > > > > >> > > > "sub-stream" will continue being possible, now via a > > > single > > > > PR > > > > > > to > > > > > > > >> > > > `kie-tools`. Cross-repo PRs between Category A and > > > Category > > > > B > > > > > > will > > > > > > > >> > > > continue not being possible, and a 1-week delay > between > > > > > merging > > > > > > > >> > > > something on Category A and using it on Category B > will > > > > still > > > > > > > >> happen. > > > > > > > >> > > > > > > > > > > >> > > > It's worth mentioning that `kie-tools`, however, does > > > allow > > > > > for > > > > > > > >> sparse > > > > > > > >> > > > checkouts and partial builds, so working with a subset > > of > > > > the > > > > > > > >> monorepo > > > > > > > >> > > > is possible and encouraged. Making changes only to > > > > > > > >> > > > `packages/kn-plugin-workflow`, for example, will have > > the > > > PR > > > > > > > checks > > > > > > > >> > > > run in < 10 minutes, as you can see here: > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-tools/actions/runs/8237244382/job/22525511722?pr=2136 > > > > > > > >> > > > . > > > > > > > >> > > > We're not compromising when running partial builds > too. > > We > > > > > know > > > > > > > that > > > > > > > >> > > > the entire repo will continue working even after only > > > > > building a > > > > > > > >> small > > > > > > > >> > > > subset of the changes. Doing partial or full builds is > > > > > > > automatically > > > > > > > >> > > > determined by the changes of a PR. > > > > > > > >> > > > > > > > > > > >> > > > Keep in mind that, even though I'm proposing we move a > > > bunch > > > > > of > > > > > > > >> > > > additional stuff into `kie-tools`, I see this as a > > > TEMPORARY > > > > > > > >> solution > > > > > > > >> > > > for our codebase. `kie-tools` would host some > additional > > > > stuff > > > > > > > >> > > > TEMPORARILY so that we can release and continue moving > > > > > forward. > > > > > > > >> > > > > > > > > > > >> > > > As I mentioned on other places, `kie-tools` became a > > > > polyglot > > > > > > > >> monorepo > > > > > > > >> > > > out of necessity, and although I'm really proud of > what > > we > > > > > > > achieved > > > > > > > >> > > > there so far, I don't think `kie-tools` has a setup > that > > > is > > > > > > > suitable > > > > > > > >> > > > for all the different nuances that compose our > > community. > > > > I'm > > > > > > well > > > > > > > >> > > > aware that a polyglot monorepo that does not follow > > > > widespread > > > > > > > >> > > > conventions will scare some people away, and as much > as > > > > we've > > > > > > > tried > > > > > > > >> to > > > > > > > >> > > > make build instructions clear, we can't always get > past > > > the > > > > > > > >> prejudice > > > > > > > >> > > > some people have towards the "front-end" ecosystem. > > > > > > > >> > > > > > > > > > > >> > > > With all that said, I keep thinking this is the best > > > course > > > > of > > > > > > > >> action > > > > > > > >> > > > for us right now. We keep most of our stuff unchanged, > > we > > > > > > unblock > > > > > > > >> the > > > > > > > >> > > > release, and we have a working setup that will suit us > > > well > > > > > > while > > > > > > > we > > > > > > > >> > > > discuss and reach a conclusion regarding the future of > > our > > > > > > > codebase > > > > > > > >> > > > structure. > > > > > > > >> > > > > > > > > > > >> > > > Let me paint a quick picture here of what our code > base > > > > would > > > > > > look > > > > > > > >> > > > like, repository-wise, if my proposal is accepted: > > > > > > > >> > > > > > > > > > > >> > > > CATEGORY REPO > > > > > > > >> > > > ===================== > > > > > > > >> > > > A incubator-kie-kogito-pipelines > > > > > > > >> > > > A incubator-kie-drools > > > > > > > >> > > > A incubator-kie-optaplanner > > > > > > > >> > > > A incubator-kie-optaplanner-quickstarts > > > > > > > >> > > > A incubator-kie-kogito-runtimes > > > > > > > >> > > > A incubator-kie-kogito-apps > > > > > > > >> > > > A incubator-kie-kogito-examples > > > > > > > >> > > > A incubator-kie-kogito-images > > > > > > > >> > > > A incubator-kie-kogito-docs > > > > > > > >> > > > A incubator-kie-kogito-benchmarks > > > > > > > >> > > > A incubator-kie-docs > > > > > > > >> > > > A incubator-kie-benchmarks > > > > > > > >> > > > ===================== > > > > > > > >> > > > B incubator-kie-sandbox-quarkus-accelerator > > > > > > > >> > > > B incubator-kie-tools > > > > > > > >> > > > ===================== > > > > > > > >> > > > D incubator-kie-kogito-operator > > > > > > > >> > > > ===================== > > > > > > > >> > > > E incubator-kie-issues > > > > > > > >> > > > E incubator-kie-kogito-website > > > > > > > >> > > > E incubator-kie-website > > > > > > > >> > > > E incubator-kie-kogito-online > > > > > > > >> > > > E incubator-kie-kogito-online-staging > > > > > > > >> > > > ===================== > > > > > > > >> > > > > > > > > > > >> > > > * Category C becomes part of Category A, and > > > > > > > >> > > > `kogito-serverless-operator` moves entirely inside > > > > > `kie-tools`. > > > > > > > >> > > > * With `kogito-swf-{builder,devmode}` images and > > > > > > > >> > > > `kogito-serverless-operator` inside `kie-tools`, there > > are > > > > no > > > > > > > cycles > > > > > > > >> > > > anymore, as inside `kie-tools`, we can granularly > build: > > > > > > > >> > > > 1. packages/sonataflow-deployment-webapp > > > > > > > >> > > > 2. packages/sonataflow-quarkus-devui > > > > > > > >> > > > 3. packages/sonataflow-images (containing > > > > > > `kogito-swf-builder` > > > > > > > >> and > > > > > > > >> > > > `kogito-swf-devmode`) > > > > > > > >> > > > 4. packages/sonataflow-operator (contents from > > > > > > > >> > > > `kogito-serverless-operator`) > > > > > > > >> > > > 5. packages/kn-plugin-sonataflow > > > > > > > (`packages/kn-plugin-workflow`, > > > > > > > >> > > > but renamed) > > > > > > > >> > > > > > > > > > > >> > > > The second part of the proposal is the release process > > > > itself, > > > > > > > >> > > > assuming the structure above is what we have. > > > > > > > >> > > > > > > > > > > >> > > > Here it is: > > > > > > > >> > > > > > > > > > > >> > > > 1. Define a timestamped SNAPSHOT to be used as cutting > > > point > > > > > for > > > > > > > >> > > > Category A repos. > > > > > > > >> > > > 2. Update Category B repos to point to this > timestamped > > > > > > SNAPSHOT, > > > > > > > >> and > > > > > > > >> > > > verify that everything is working. > > > > > > > >> > > > 3. At this point, with everything working, we can > branch > > > out > > > > > to > > > > > > > >> > > > `10.0.x`. Category A from the timestamped SNAPSHOT > tag, > > > and > > > > > > > >> Category B > > > > > > > >> > > > from `main`. > > > > > > > >> > > > 4. All Category A and Category B repos update their > > > versions > > > > > to > > > > > > > >> > > > 10.0.0, in their `10.0.x` branches. > > > > > > > >> > > > 5. Update Category B repos to point to Category A > repos > > > > using > > > > > > the > > > > > > > >> > > > 10.0.0 version. > > > > > > > >> > > > 6. At this point, we can vote on the release based on > > the > > > > > > `10.0.x` > > > > > > > >> > > > branches, given we don't expect any code changes > > anymore. > > > > > > > >> > > > 7. After voting passes, we're good to start the > release > > > > > process. > > > > > > > >> > > > 8. Category A repos follow their manual/automated > > release > > > > > > process, > > > > > > > >> > > > pointing to the `10.0.x` branch. Tags pushed to Git, > and > > > > built > > > > > > > >> > > > artifacts pushed to their registries. > > > > > > > >> > > > 9. We wait a little bit for Category A artifacts to be > > > > > > propagated > > > > > > > on > > > > > > > >> > > > registries. ~1 day. > > > > > > > >> > > > 10. Category B repos follow their manual/automated > > release > > > > > > > process, > > > > > > > >> > > > pointing to the `10.0.x` branch. Tags pushed to Git, > and > > > > built > > > > > > > >> > > > artifacts pushed to their registries. > > > > > > > >> > > > 11. Category D repos are ignored. > > > > > > > >> > > > 12. Category E repos can be manually tagged with > 10.0.0 > > > from > > > > > > their > > > > > > > >> > > > default branches. > > > > > > > >> > > > > > > > > > > >> > > > More needs to be discussed if we're planning to > maintain > > > > > > multiple > > > > > > > >> > > > release streams in parallel, but I guess it can wait > for > > > > after > > > > > > > >> Apache > > > > > > > >> > > > KIE 10. > > > > > > > >> > > > > > > > > > > >> > > > Thank you for reading, and I'm looking forward to > > hearing > > > > back > > > > > > > from > > > > > > > >> > > > everyone. > > > > > > > >> > > > > > > > > > > >> > > > Of course, alternative solutions are possible. This > > email, > > > > > > > however, > > > > > > > >> > > > summarizes my view of how we should attack the > problem, > > > > > > > considering > > > > > > > >> > > > disruption, required effort, the release process > itself, > > > and > > > > > > > >> history. > > > > > > > >> > > > Feel free to propose alternatives. This is not a > voting > > > > > thread. > > > > > > > >> > > > > > > > > > > >> > > > Regards, > > > > > > > >> > > > > > > > > > > >> > > > Tiago Bento > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > --------------------------------------------------------------------- > > > > > > > >> > > > To unsubscribe, e-mail: > dev-unsubscr...@kie.apache.org > > > > > > > >> > > > For additional commands, e-mail: > > dev-h...@kie.apache.org > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
