Hi Tiago Thanks for the proposal, which I think is one that deserves careful consideration and some discussion before getting voted, but I can anticipate it is workable from my perspective, so I'm not going to oppose it as I did with the previous one (which is public I considered not feasible) Since the root cause behind all these discussion is the mono/multi repo question, let me share my thought about it When Kogito was initially designed ( I was not here, but doing some historical inference ;)) we were in the era of "Microservice for everything" (a reflection of Netflix success on using this kind of architecture) or, in other terms, "Kill the monolith" (a reflection on how science computer guys tend to adopt fashion in a black-white way) Although the repo structure of a project does not necessarily resemble the architecture view, it tends to do so, and Kogito was no exception... partially. The runtimes microservice has its own repo (kogito-runtimes), but it was kind of split (for historical reasons) into one additional repo, containing drools stuff (which acts as a library for runtimes) At the same time, we have the kogito-app repo, which depends on runtime (and hence in drools too) that holds several microservices: data-index, data-audit, job-services... and trusty (which I feel should be removed eventually). If we were being strict about mapping one microservice to one repo, every one of them will be in their own repo (with another repo containing the common libraries shared by all of them) In a sense, the backend repositories kind of follow the philosophy you are now proposing: one repo for the core microservice (runtimes), another repo for the rest of the stuff (apps) and drools (because of historical reason) with a repo of its own. Over that, there were some additional repos: kie docs (which is actually divided into at least two repos holding documentation ;)) and kie examples. And of course, the repo you are more familiar with, the tools one. The current situation is good (although not optimal) for a developer willing to contribute to a particular area. Currently, drools guys just work on drools (and sometime take a look at runtimes it something gets broken), runtimes guys like me hardly might look into drools repo and eventually contribute is some areas of apps (in may case, data-index), so the stuff that we currently check out in our local machines is reasonable. That won't really change if we have a backend repo containing everything (I guess most of us have checked out drools, runtimes and apps in our local envs) . However, I feel we lost the proper separation of concerns which, in any case, more than in the repo structure depends on us, which are supposed to be smart guys aware of the different responsibilities of the different components, not mixing them in the code (Though, this not always happened, for example with the Quarkus and springboot stuff) After this quick historical survey, since it is clear that we are architecturally tending to a monolith (for example, we are promoting embedded data index and data index in same db than runtimes), it is logical that you tend also to coalesce repos (which, again, are already partially coalesced) into two: backend (which you called Java) and frontend (which you called tools) and I do not think there are strong arguments to prevent that move (it won't be my preferred option, I think we should have one repo for common stuff to all engines and one repo per engine, but this is the architect inside me talking ;)) However, I still feel that docs and examples should be kept in separate repos. Why keep them separate? Because I feel there might be a lot of guys out there that will be happy just reading the docs and testing the examples, without having to check out the backend or frontend in their computers. Which is true for users is also true for contributors. It is true that most of the contribution to docs and examples is done by the developers that actually wrote frontend and backend code, but it is also true that usually there are (or should be) "doc specialists" that just work on examples readmes and in docs. However, this example/docs thing is a minor concern (it would be an extra step to the proposal, once you have kie and extras in place, it should not be that hard to set up an example repo that depends on the other two) . Anyway, as a summary of this long e-mail, I think having two repos storing the source code is a consistent approach. +0
On Tue, Feb 18, 2025 at 12:22 AM Tiago Bento <tiagobe...@apache.org> wrote: > Hi everyone, > > As a final attempt to shift the focus of the Apache KIE community away > from build systems and codebase structure and to allow us to move > forward with a good release cadence while fostering innovation in the > Open-Source Business Automation space, Alex and I wrote what we hope > to be the last infrastructure-related proposal from us (for quite a > while, at least). > > We sincerely believe that, if approved, this proposal will put Apache > KIE in a very advantageous place to welcome new contributors, > encourage current contributors to do their best work yet and multiply > the reach of our technology to even more users. We hope everyone > reading this feels heard as we try to incorporate all the feedback we > gathered since our first day inside Apache. > > We opted to go for a [VOTE] directly to avoid risking an endless > conversation again. While we understand this is not the preferred > approach for contributing, this subject is still sensitive enough for > us to have chosen this method, hopefully for the last time. > > Tiago and Alex. > > # > # Problem scope > > Recently during the many threads that were raised in this mailing > list, it became clear that there’s a general concern about the > direction of the codebase structure and a continued feedback against > the monorepo-ization of the Apache KIE codebase, which seems to be > stealing our energy and making it impossible to focus on what we all > do best: features and innovation. > > It’s also clear that the current infrastructure hasn’t been helpful. > The level of complexity necessary to maintain the current repositories > is holding us back from a more streamlined approach to PR checks and > CI and, most importantly, to our releases, preventing us from > maintaining a good cadence. > > This proposal brings a compromise to the table, between the current > poly-repo and a monorepo: a duo-repo, with one repo named `kie`, > hosting all Java runtime code, and another repo named `kie-extras`, > hosting everything else (including Docs, Examples, Websites, Container > images, Editors, Web apps, Quakus Dev UIs, Jenkins release jobs etc). > > The duo-repo-ization of the Apache KIE codebase will allow us to > dramatically simplify the automations we have for PR checks, CI and > release jobs as well as the operation for daily contributions, > stopping the dependence we have on SNAPSHOTs, for example. The > ultimate goal is to replace the complicated `build-chain`-based PR > checks and the custom Jenkins framework we have for CI/releases. > > # > # What is not in scope > > - Cross-repo PRs; as it leads to systems similar to `build-chain` > and the custom Jenkins framework. We want to avoid going in that > direction again. > - Sparse-checkouts mechanism for `kie`; Due to the complexity of > setting up something like this, and the other mechanisms that exist to > mitigate the increase in size of the repo, this won’t be pursued > initially. > - A complete review of all repositories under `apache/incubator-kie-*` > > # > # The proposal > > Changes to repositories: > - `drools` is: > - Renamed to `kie`, keeping the most-starred repository we have on > GitHub. > - Updated to include `optaplanner`, `kogito-runtimes` and > `kogito-apps`; preserving Git history. > - New top-level modules will be created for each repository, > making things simpler at the beginning. > - Top-level modules prefixed with `drools-` go to a new > folder called `drools`, and top-level modules prefixed with `kie-` go > to a new folder called `kie`. > - `kie-tools` is: > - Renamed to `kie-extras`; reflecting its multi-purposed nature > of hosting a variety of different package types > - `optaplanner`, `kogito-runtimes`, and `kogito-apps` are: > - Updated with READMEs/descriptions pointing to the new `kie` repo > - Archived by Apache INFRA once `kie` is fully operational > - `kogito-pipelines` is: > - Updated with a disclaimer on its README saying it’s no longer used. > - Archived by Apache INFRA once `kie` is fully operational > > Changes to automations: > - Jenkins is almost completely reset, except for a few jobs > necessary for the Release Procedure [1]. > - Timestamped SNAPSHOTs stop being published every Sunday. > - Regular SNAPSHOTs stop being published every day. > - `apache.snapshots` repository is cleaned up so that no SNAPSHOTs > are available anymore. > - PRs sent to `kie` will be checked using a new GitHub Actions > hosted inside it and WON’T build `kie-extras` > - PRs sent to `kie-extras` will be checked as they already are, with > a new step at the beginning that will build (or fetch from a cached > GitHub Actions artifact) `kie` at a versioned commit SHA. > - A new weekly bot is created to update the versioned `kie` SHA at > `kie-extras` via PR. Contributors to `kie-extras` start their week > taking a look at it and working together to get that merged ASAP. > - A new release job stored in `kie-extras` is created to build and > publish `kie` on a Release Candidate. > > Communication: > - Apache KIE’s website and documentation are updated to reflect the > new duo-repo structure. > > ### Impact on operation and contributions > > - Contributors to `optaplanner`, `kogito-runtimes` and `kogito-apps` > will move to `kie` > - PR checks will take the same time (or less, if we’re able to > capitalize more on parallel builds, given every module is going to be > part of the same build in Maven’s reactor). > - Unified Maven dependency management, less surface to manage > external dependencies. > - Contributions to `kie-extras` that depend on `kie` will continue > being bi-phased, the same way they are today with the timestamped > SNAPSHOTs. > > ### Notes on SNAPSHOTs > > - According to Apache’s guidelines on unreleased artifacts, we > should be very conscious about our SNAPSHOT usage in regards to > encouraging users to consume them or even to depend on them. See > https://infra.apache.org/release-distribution.html#unreleased > - Users will always have access to the latest release and will be > encouraged to step on the development itself to access the bleeding > edge (Building 999-SNAPSHOT locally from `kie` @ `main`). > - Given the discrepancy between `kie` and `kie-extras`, to make CI > simpler and not delve into cross-repo PR systems, it would be > confusing to have some SNAPSHOTs published from `kie-extras` that are > not aligned with SNAPSHOTs from `kie`. > - Given the simplified structure of the codebase, publishing > SNAPSHOTs would encourage usage that’s either not aligned with Apache > or with our strategy of not having cross-repo PRs. We can revisit that > in the future if we find a justification for such. This proposal > doesn’t want to get in the way of any future work, just establish a > comprehensive baseline for us to evolve on top of. > > # > # Actionable items (plan) > > Phase 1 (New structure) > - A new GitHub Actions workflow is created on `drools` for PR checks > (without building any other repos) > - `optaplanner` is incorporated as a top-level folder on `drools`, > preserving Git history > - `optaplanner`’s README is updated, pointing to `drools` as its new > home. Contributions to it are frozen. > - `kogito-runtimes` is incorporated as a top-level folder on > `drools`, preserving Git history > - `kogito-runtimes`’s README is updated, pointing to `drools` as its > new home. Contributions to it are frozen. > - `kogito-apps` is incorporated as a top-level folder on `drools`, > preserving Git history > - `kogito-apps`’s README is updated, pointing to `drools` as its new > home. Contributions to it are frozen. > - `drools` is renamed to `kie` (GitHub redirects will make sure > users accessing/cloning `drools` automatically go to `kie`) > - `kie`’s README is updated to reflect its new top-level packages. > - `kie-tools` is renamed to `kie-extras`. > - `kie-tools` has its reference to `kie` updated to a versioned > commit SHA, at the same time as its PR checks and CI are updated to > build `kie` prior to building itself. > > Phase 2 (Cleanup) > - `build-chain`-based GitHub Actions are deleted from all repositories. > - The custom Jenkins framework jobs are removed from Jenkins > - Update Documentation and Websites to reflect new codebase structure > > Phase 3 (Release) > - A new release job is created on Jenkins (hosted on `kie-extras`) > to release Maven modules from `kie`. > - The Release Procedure for 10.2 is updated to accommodate the new > codebase structure > > Phase 4 (Archive Repos) > - kogito-pipelines, optaplanner, kogito-runtimes, and kogito-apps > are archived by INFRA > > # > # Commitment and rough timeline > > Alex and Tiago are fully committed to executing this proposal with a > start date right after 10.1.0 is fully released (approved by IPMC and > published to all external registries). > > The rough estimate to execute is about 4 weeks; with [HEADS UP]s > coming to the mailing list whenever a change approaches. > > 10.2.0 will be the first release already with the new structure, where > the new release jobs will be validated. > > > [1] > https://docs.google.com/document/d/15V-1e62bmmWeOdFUnSJHymPYoeb0TTwp0bhxPuJn4zo > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > For additional commands, e-mail: dev-h...@kie.apache.org > >
