Hi everyone,
As a final attempt to shift the focus of the Apache KIE community away
from build systems and codebase structure and to allow us to move
forward with a good release cadence while fostering innovation in the
Open-Source Business Automation space, Alex and I wrote what we hope
to be the last infrastructure-related proposal from us (for quite a
while, at least).
We sincerely believe that, if approved, this proposal will put Apache
KIE in a very advantageous place to welcome new contributors,
encourage current contributors to do their best work yet and multiply
the reach of our technology to even more users. We hope everyone
reading this feels heard as we try to incorporate all the feedback we
gathered since our first day inside Apache.
We opted to go for a [VOTE] directly to avoid risking an endless
conversation again. While we understand this is not the preferred
approach for contributing, this subject is still sensitive enough for
us to have chosen this method, hopefully for the last time.
Tiago and Alex.
#
# Problem scope
Recently during the many threads that were raised in this mailing
list, it became clear that there’s a general concern about the
direction of the codebase structure and a continued feedback against
the monorepo-ization of the Apache KIE codebase, which seems to be
stealing our energy and making it impossible to focus on what we all
do best: features and innovation.
It’s also clear that the current infrastructure hasn’t been helpful.
The level of complexity necessary to maintain the current repositories
is holding us back from a more streamlined approach to PR checks and
CI and, most importantly, to our releases, preventing us from
maintaining a good cadence.
This proposal brings a compromise to the table, between the current
poly-repo and a monorepo: a duo-repo, with one repo named `kie`,
hosting all Java runtime code, and another repo named `kie-extras`,
hosting everything else (including Docs, Examples, Websites, Container
images, Editors, Web apps, Quakus Dev UIs, Jenkins release jobs etc).
The duo-repo-ization of the Apache KIE codebase will allow us to
dramatically simplify the automations we have for PR checks, CI and
release jobs as well as the operation for daily contributions,
stopping the dependence we have on SNAPSHOTs, for example. The
ultimate goal is to replace the complicated `build-chain`-based PR
checks and the custom Jenkins framework we have for CI/releases.
#
# What is not in scope
- Cross-repo PRs; as it leads to systems similar to `build-chain`
and the custom Jenkins framework. We want to avoid going in that
direction again.
- Sparse-checkouts mechanism for `kie`; Due to the complexity of
setting up something like this, and the other mechanisms that exist to
mitigate the increase in size of the repo, this won’t be pursued
initially.
- A complete review of all repositories under `apache/incubator-kie-*`
#
# The proposal
Changes to repositories:
- `drools` is:
- Renamed to `kie`, keeping the most-starred repository we have on GitHub.
- Updated to include `optaplanner`, `kogito-runtimes` and
`kogito-apps`; preserving Git history.
- New top-level modules will be created for each repository,
making things simpler at the beginning.
- Top-level modules prefixed with `drools-` go to a new
folder called `drools`, and top-level modules prefixed with `kie-` go
to a new folder called `kie`.
- `kie-tools` is:
- Renamed to `kie-extras`; reflecting its multi-purposed nature
of hosting a variety of different package types
- `optaplanner`, `kogito-runtimes`, and `kogito-apps` are:
- Updated with READMEs/descriptions pointing to the new `kie` repo
- Archived by Apache INFRA once `kie` is fully operational
- `kogito-pipelines` is:
- Updated with a disclaimer on its README saying it’s no longer used.
- Archived by Apache INFRA once `kie` is fully operational
Changes to automations:
- Jenkins is almost completely reset, except for a few jobs
necessary for the Release Procedure [1].
- Timestamped SNAPSHOTs stop being published every Sunday.
- Regular SNAPSHOTs stop being published every day.
- `apache.snapshots` repository is cleaned up so that no SNAPSHOTs
are available anymore.
- PRs sent to `kie` will be checked using a new GitHub Actions
hosted inside it and WON’T build `kie-extras`
- PRs sent to `kie-extras` will be checked as they already are, with
a new step at the beginning that will build (or fetch from a cached
GitHub Actions artifact) `kie` at a versioned commit SHA.
- A new weekly bot is created to update the versioned `kie` SHA at
`kie-extras` via PR. Contributors to `kie-extras` start their week
taking a look at it and working together to get that merged ASAP.
- A new release job stored in `kie-extras` is created to build and
publish `kie` on a Release Candidate.
Communication:
- Apache KIE’s website and documentation are updated to reflect the
new duo-repo structure.
### Impact on operation and contributions
- Contributors to `optaplanner`, `kogito-runtimes` and `kogito-apps`
will move to `kie`
- PR checks will take the same time (or less, if we’re able to
capitalize more on parallel builds, given every module is going to be
part of the same build in Maven’s reactor).
- Unified Maven dependency management, less surface to manage
external dependencies.
- Contributions to `kie-extras` that depend on `kie` will continue
being bi-phased, the same way they are today with the timestamped
SNAPSHOTs.
### Notes on SNAPSHOTs
- According to Apache’s guidelines on unreleased artifacts, we
should be very conscious about our SNAPSHOT usage in regards to
encouraging users to consume them or even to depend on them. See
https://infra.apache.org/release-distribution.html#unreleased
- Users will always have access to the latest release and will be
encouraged to step on the development itself to access the bleeding
edge (Building 999-SNAPSHOT locally from `kie` @ `main`).
- Given the discrepancy between `kie` and `kie-extras`, to make CI
simpler and not delve into cross-repo PR systems, it would be
confusing to have some SNAPSHOTs published from `kie-extras` that are
not aligned with SNAPSHOTs from `kie`.
- Given the simplified structure of the codebase, publishing
SNAPSHOTs would encourage usage that’s either not aligned with Apache
or with our strategy of not having cross-repo PRs. We can revisit that
in the future if we find a justification for such. This proposal
doesn’t want to get in the way of any future work, just establish a
comprehensive baseline for us to evolve on top of.
#
# Actionable items (plan)
Phase 1 (New structure)
- A new GitHub Actions workflow is created on `drools` for PR checks
(without building any other repos)
- `optaplanner` is incorporated as a top-level folder on `drools`,
preserving Git history
- `optaplanner`’s README is updated, pointing to `drools` as its new
home. Contributions to it are frozen.
- `kogito-runtimes` is incorporated as a top-level folder on
`drools`, preserving Git history
- `kogito-runtimes`’s README is updated, pointing to `drools` as its
new home. Contributions to it are frozen.
- `kogito-apps` is incorporated as a top-level folder on `drools`,
preserving Git history
- `kogito-apps`’s README is updated, pointing to `drools` as its new
home. Contributions to it are frozen.
- `drools` is renamed to `kie` (GitHub redirects will make sure
users accessing/cloning `drools` automatically go to `kie`)
- `kie`’s README is updated to reflect its new top-level packages.
- `kie-tools` is renamed to `kie-extras`.
- `kie-tools` has its reference to `kie` updated to a versioned
commit SHA, at the same time as its PR checks and CI are updated to
build `kie` prior to building itself.
Phase 2 (Cleanup)
- `build-chain`-based GitHub Actions are deleted from all repositories.
- The custom Jenkins framework jobs are removed from Jenkins
- Update Documentation and Websites to reflect new codebase structure
Phase 3 (Release)
- A new release job is created on Jenkins (hosted on `kie-extras`)
to release Maven modules from `kie`.
- The Release Procedure for 10.2 is updated to accommodate the new
codebase structure
Phase 4 (Archive Repos)
- kogito-pipelines, optaplanner, kogito-runtimes, and kogito-apps
are archived by INFRA
#
# Commitment and rough timeline
Alex and Tiago are fully committed to executing this proposal with a
start date right after 10.1.0 is fully released (approved by IPMC and
published to all external registries).
The rough estimate to execute is about 4 weeks; with [HEADS UP]s
coming to the mailing list whenever a change approaches.
10.2.0 will be the first release already with the new structure, where
the new release jobs will be validated.
[1]
https://docs.google.com/document/d/15V-1e62bmmWeOdFUnSJHymPYoeb0TTwp0bhxPuJn4zo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org
For additional commands, e-mail: dev-h...@kie.apache.org
