I forgot to mention, in my previous email, that my preferred repo structure, one repo to store the common stuff that acts as library for the different engines and one repo per each engine, where engine means: rules, bpmn process, swf process and so on, is simple from CI perspective. When modifying the engine repo, we will just compile the content of the engine repo being modified (optionally, we might compile the common stuff). That will be far faster than the current approach (when you change SWF, you just check SWF, when you change BPMN, you just check BPMn, and so on). Currently, when we modify something in runtimes, we are compiling and testing: codegen for all engines, bpmn, swf, quarkus, sprinboot, all apps and all examples (we are not compiling drools though). With one kie repo, we will compile and test everything (that will take a substantial amount of time and well, the red is there not because of the multi repos, but because of some flaky test that is unrelated with the stuff you are modifying, so thinking that coalescing or separating repos fixes the unrelated red is wishful thinking, I bet that it will make it worse for drools folks, which currently are not slowed down by keycloak flaky test) I know my proposal is not feasible because of historical reasons/lack of resources, but I think it was worth writing about, because it will really fix the "red flag" problem in most PRs ;)
On Tue, Feb 18, 2025 at 12:04 PM Toni Rikkola <rikk...@apache.org> wrote: > Right. I have been informed that this change will not introduce the > problem I am describing, since it already exists. > > That means we are already in trouble, but this proposal does not increase > them. > > +1 > > Toni > > On 2025/02/18 09:56:38 Toni Rikkola wrote: > > -1 > > I should use the Apache account for votes to count. > > > > Toni > > > > On 2025/02/18 09:18:20 Toni Rikkola wrote: > > > I have very high doubts about this working and I really hope I > understood > > > this wrong. > > > > > > PRs sent to `kie` will be checked using a new GitHub Actions > > > > hosted inside it and WON’T build `kie-extras` > > > > > > > > > A new weekly bot is created to update the versioned `kie` SHA at > > > > `kie-extras` via PR. Contributors to `kie-extras` start their week > > > > taking a look at it and working together to get that merged ASAP. > > > > > > > > > Each merged PR should have our code base in a state that works from > top to > > > bottom. > > > This is people working on kie pushing work for those working on > kie-extras. > > > Kie dev/vendor is losing time and/or money for the kie-extra devs and > > > vendors. > > > We are independent developers from the community perspective. When the > kie > > > PR is merged there is a task for someone to do and no requirement for > > > anyone to do it. > > > There isn't even a requirement for anyone to revert any commit. So now > > > kie-extras dev needs to touch kie and undo work that was already > approved > > > or not do that and we fight about who fixes it. > > > There is no guarantee the change done at kie is fixable in kie-extras. > Then > > > it needs to be reverted in kie and that takes a week. The repositories > will > > > be out of sync for 2 weeks. > > > > > > I do like the idea of reducing the repositories. Most bug fixes I used > to > > > do hit 3-4 repositories and I spent weeks restarting the PR builds > every > > > workday to get a full green. With this cycle even a 2 repo change can > take > > > months. > > > > > > -1. > > > > > > Toni > > > > > > On Tue, Feb 18, 2025 at 10:34 AM Kennedy Bowers <kbow...@apache.org> > wrote: > > > > > > > +1 > > > > > > > > On 2025/02/18 07:53:06 Yeser Amer wrote: > > > > > +1 > > > > > > > > > > Thank you. > > > > > > > > > > On 2025/02/17 23:23:14 Alex Porcelli wrote: > > > > > > +1 > > > > > > > > > > > > On Mon, Feb 17, 2025 at 6:22 PM Tiago Bento < > tiagobe...@apache.org> > > > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > As a final attempt to shift the focus of the Apache KIE > community > > > > away > > > > > > > from build systems and codebase structure and to allow us to > move > > > > > > > forward with a good release cadence while fostering innovation > in the > > > > > > > Open-Source Business Automation space, Alex and I wrote what > we hope > > > > > > > to be the last infrastructure-related proposal from us (for > quite a > > > > > > > while, at least). > > > > > > > > > > > > > > We sincerely believe that, if approved, this proposal will put > Apache > > > > > > > KIE in a very advantageous place to welcome new contributors, > > > > > > > encourage current contributors to do their best work yet and > multiply > > > > > > > the reach of our technology to even more users. We hope > everyone > > > > > > > reading this feels heard as we try to incorporate all the > feedback we > > > > > > > gathered since our first day inside Apache. > > > > > > > > > > > > > > We opted to go for a [VOTE] directly to avoid risking an > endless > > > > > > > conversation again. While we understand this is not the > preferred > > > > > > > approach for contributing, this subject is still sensitive > enough for > > > > > > > us to have chosen this method, hopefully for the last time. > > > > > > > > > > > > > > Tiago and Alex. > > > > > > > > > > > > > > # > > > > > > > # Problem scope > > > > > > > > > > > > > > Recently during the many threads that were raised in this > mailing > > > > > > > list, it became clear that there’s a general concern about the > > > > > > > direction of the codebase structure and a continued feedback > against > > > > > > > the monorepo-ization of the Apache KIE codebase, which seems > to be > > > > > > > stealing our energy and making it impossible to focus on what > we all > > > > > > > do best: features and innovation. > > > > > > > > > > > > > > It’s also clear that the current infrastructure hasn’t been > helpful. > > > > > > > The level of complexity necessary to maintain the current > > > > repositories > > > > > > > is holding us back from a more streamlined approach to PR > checks and > > > > > > > CI and, most importantly, to our releases, preventing us from > > > > > > > maintaining a good cadence. > > > > > > > > > > > > > > This proposal brings a compromise to the table, between the > current > > > > > > > poly-repo and a monorepo: a duo-repo, with one repo named > `kie`, > > > > > > > hosting all Java runtime code, and another repo named > `kie-extras`, > > > > > > > hosting everything else (including Docs, Examples, Websites, > > > > Container > > > > > > > images, Editors, Web apps, Quakus Dev UIs, Jenkins release > jobs etc). > > > > > > > > > > > > > > The duo-repo-ization of the Apache KIE codebase will allow us > to > > > > > > > dramatically simplify the automations we have for PR checks, > CI and > > > > > > > release jobs as well as the operation for daily contributions, > > > > > > > stopping the dependence we have on SNAPSHOTs, for example. The > > > > > > > ultimate goal is to replace the complicated > `build-chain`-based PR > > > > > > > checks and the custom Jenkins framework we have for > CI/releases. > > > > > > > > > > > > > > # > > > > > > > # What is not in scope > > > > > > > > > > > > > > - Cross-repo PRs; as it leads to systems similar to > `build-chain` > > > > > > > and the custom Jenkins framework. We want to avoid going in > that > > > > > > > direction again. > > > > > > > - Sparse-checkouts mechanism for `kie`; Due to the > complexity of > > > > > > > setting up something like this, and the other mechanisms that > exist > > > > to > > > > > > > mitigate the increase in size of the repo, this won’t be > pursued > > > > > > > initially. > > > > > > > - A complete review of all repositories under > > > > `apache/incubator-kie-*` > > > > > > > > > > > > > > # > > > > > > > # The proposal > > > > > > > > > > > > > > Changes to repositories: > > > > > > > - `drools` is: > > > > > > > - Renamed to `kie`, keeping the most-starred repository > we > > > > have on > > > > > > > GitHub. > > > > > > > - Updated to include `optaplanner`, `kogito-runtimes` and > > > > > > > `kogito-apps`; preserving Git history. > > > > > > > - New top-level modules will be created for each > > > > repository, > > > > > > > making things simpler at the beginning. > > > > > > > - Top-level modules prefixed with `drools-` go to a > new > > > > > > > folder called `drools`, and top-level modules prefixed with > `kie-` go > > > > > > > to a new folder called `kie`. > > > > > > > - `kie-tools` is: > > > > > > > - Renamed to `kie-extras`; reflecting its multi-purposed > nature > > > > > > > of hosting a variety of different package types > > > > > > > - `optaplanner`, `kogito-runtimes`, and `kogito-apps` are: > > > > > > > - Updated with READMEs/descriptions pointing to the new > `kie` > > > > repo > > > > > > > - Archived by Apache INFRA once `kie` is fully > operational > > > > > > > - `kogito-pipelines` is: > > > > > > > - Updated with a disclaimer on its README saying it’s no > > > > longer used. > > > > > > > - Archived by Apache INFRA once `kie` is fully > operational > > > > > > > > > > > > > > Changes to automations: > > > > > > > - Jenkins is almost completely reset, except for a few jobs > > > > > > > necessary for the Release Procedure [1]. > > > > > > > - Timestamped SNAPSHOTs stop being published every Sunday. > > > > > > > - Regular SNAPSHOTs stop being published every day. > > > > > > > - `apache.snapshots` repository is cleaned up so that no > SNAPSHOTs > > > > > > > are available anymore. > > > > > > > - PRs sent to `kie` will be checked using a new GitHub > Actions > > > > > > > hosted inside it and WON’T build `kie-extras` > > > > > > > - PRs sent to `kie-extras` will be checked as they already > are, > > > > with > > > > > > > a new step at the beginning that will build (or fetch from a > cached > > > > > > > GitHub Actions artifact) `kie` at a versioned commit SHA. > > > > > > > - A new weekly bot is created to update the versioned `kie` > SHA at > > > > > > > `kie-extras` via PR. Contributors to `kie-extras` start their > week > > > > > > > taking a look at it and working together to get that merged > ASAP. > > > > > > > - A new release job stored in `kie-extras` is created to > build and > > > > > > > publish `kie` on a Release Candidate. > > > > > > > > > > > > > > Communication: > > > > > > > - Apache KIE’s website and documentation are updated to > reflect the > > > > > > > new duo-repo structure. > > > > > > > > > > > > > > ### Impact on operation and contributions > > > > > > > > > > > > > > - Contributors to `optaplanner`, `kogito-runtimes` and > > > > `kogito-apps` > > > > > > > will move to `kie` > > > > > > > - PR checks will take the same time (or less, if we’re able > to > > > > > > > capitalize more on parallel builds, given every module is > going to be > > > > > > > part of the same build in Maven’s reactor). > > > > > > > - Unified Maven dependency management, less surface to manage > > > > > > > external dependencies. > > > > > > > - Contributions to `kie-extras` that depend on `kie` will > continue > > > > > > > being bi-phased, the same way they are today with the > timestamped > > > > > > > SNAPSHOTs. > > > > > > > > > > > > > > ### Notes on SNAPSHOTs > > > > > > > > > > > > > > - According to Apache’s guidelines on unreleased artifacts, > we > > > > > > > should be very conscious about our SNAPSHOT usage in regards to > > > > > > > encouraging users to consume them or even to depend on them. > See > > > > > > > https://infra.apache.org/release-distribution.html#unreleased > > > > > > > - Users will always have access to the latest release and > will be > > > > > > > encouraged to step on the development itself to access the > bleeding > > > > > > > edge (Building 999-SNAPSHOT locally from `kie` @ `main`). > > > > > > > - Given the discrepancy between `kie` and `kie-extras`, to > make CI > > > > > > > simpler and not delve into cross-repo PR systems, it would be > > > > > > > confusing to have some SNAPSHOTs published from `kie-extras` > that are > > > > > > > not aligned with SNAPSHOTs from `kie`. > > > > > > > - Given the simplified structure of the codebase, publishing > > > > > > > SNAPSHOTs would encourage usage that’s either not aligned with > Apache > > > > > > > or with our strategy of not having cross-repo PRs. We can > revisit > > > > that > > > > > > > in the future if we find a justification for such. This > proposal > > > > > > > doesn’t want to get in the way of any future work, just > establish a > > > > > > > comprehensive baseline for us to evolve on top of. > > > > > > > > > > > > > > # > > > > > > > # Actionable items (plan) > > > > > > > > > > > > > > Phase 1 (New structure) > > > > > > > - A new GitHub Actions workflow is created on `drools` for PR > > > > checks > > > > > > > (without building any other repos) > > > > > > > - `optaplanner` is incorporated as a top-level folder on > `drools`, > > > > > > > preserving Git history > > > > > > > - `optaplanner`’s README is updated, pointing to `drools` as > its > > > > new > > > > > > > home. Contributions to it are frozen. > > > > > > > - `kogito-runtimes` is incorporated as a top-level folder on > > > > > > > `drools`, preserving Git history > > > > > > > - `kogito-runtimes`’s README is updated, pointing to > `drools` as > > > > its > > > > > > > new home. Contributions to it are frozen. > > > > > > > - `kogito-apps` is incorporated as a top-level folder on > `drools`, > > > > > > > preserving Git history > > > > > > > - `kogito-apps`’s README is updated, pointing to `drools` as > its > > > > new > > > > > > > home. Contributions to it are frozen. > > > > > > > - `drools` is renamed to `kie` (GitHub redirects will make > sure > > > > > > > users accessing/cloning `drools` automatically go to `kie`) > > > > > > > - `kie`’s README is updated to reflect its new top-level > packages. > > > > > > > - `kie-tools` is renamed to `kie-extras`. > > > > > > > - `kie-tools` has its reference to `kie` updated to a > versioned > > > > > > > commit SHA, at the same time as its PR checks and CI are > updated to > > > > > > > build `kie` prior to building itself. > > > > > > > > > > > > > > Phase 2 (Cleanup) > > > > > > > - `build-chain`-based GitHub Actions are deleted from all > > > > repositories. > > > > > > > - The custom Jenkins framework jobs are removed from Jenkins > > > > > > > - Update Documentation and Websites to reflect new codebase > > > > structure > > > > > > > > > > > > > > Phase 3 (Release) > > > > > > > - A new release job is created on Jenkins (hosted on > `kie-extras`) > > > > > > > to release Maven modules from `kie`. > > > > > > > - The Release Procedure for 10.2 is updated to accommodate > the new > > > > > > > codebase structure > > > > > > > > > > > > > > Phase 4 (Archive Repos) > > > > > > > - kogito-pipelines, optaplanner, kogito-runtimes, and > kogito-apps > > > > > > > are archived by INFRA > > > > > > > > > > > > > > # > > > > > > > # Commitment and rough timeline > > > > > > > > > > > > > > Alex and Tiago are fully committed to executing this proposal > with a > > > > > > > start date right after 10.1.0 is fully released (approved by > IPMC and > > > > > > > published to all external registries). > > > > > > > > > > > > > > The rough estimate to execute is about 4 weeks; with [HEADS > UP]s > > > > > > > coming to the mailing list whenever a change approaches. > > > > > > > > > > > > > > 10.2.0 will be the first release already with the new > structure, > > > > where > > > > > > > the new release jobs will be validated. > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > https://docs.google.com/document/d/15V-1e62bmmWeOdFUnSJHymPYoeb0TTwp0bhxPuJn4zo > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > > > > > > For additional commands, e-mail: dev-h...@kie.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > > > > For additional commands, e-mail: dev-h...@kie.apache.org > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > > > For additional commands, e-mail: dev-h...@kie.apache.org > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > For additional commands, e-mail: dev-h...@kie.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > For additional commands, e-mail: dev-h...@kie.apache.org > >
