Given the recent discussions about Docs [1] and Examples [2], Tiago Bento and I got together to write, in proposal form, the combination of our understanding of how we can create the baseline for content (Docs, Examples, and Websites) so that our users can consume Apache KIE releases in an expected, structured, and cohesive way. This proposal will take you < 15 min to read.
- Alex & Tiago # Problem scope There has been a significant demand for proper guidance on using Apache KIE 10.0. Users are unsure how to adopt it, as neither Documentation nor Examples were available when 10.0.0 was released. An attempt to manually adjust the examples to the 10.0.0 release has happened for a couple of weeks now, showing how time-consuming it is to defer dealing with “satellite” repositories after releases are done. Even more complicated is that well-known websites (drools.org, jbpm.org, kogito.kie.org) are outdated and out of sync with Apache KIE’s repositories and releases, thus contributing to more confusion than anything else. To avoid being unclear about why Docs, Examples, and Websites are crucial for the community's success, they are all entry points where new users can start understanding Apache KIE, how to use it and whether or not it’s a fit for them. For existing users, they are all reference material, including code snippets, institutional texts and/or vision, licenses, migration guides, and legal material. Examples, in particular, also work as playgrounds, where users can start with pre-built applications and make minor tweaks to either check if something they want is supported or simulate a development workflow on a smaller scale. Our code is worth close to nothing if there isn’t good supporting material around it. This proposal aims to amend all those problems by bringing Docs, Examples, and Websites to the Release Procedure, PR checks, and a Continuous deployment (CD) mechanism, making them all be part of our daily contributions, so they are not treated as afterthoughts. # What is not in scope - A complete restructuring of all Apache KIE repositories - Refactoring the whole CI; this proposal aims to work with the current established CI already in place - Refactor content in Docs and Example applications; this proposal aims to move stuff as-is, except when something is broken, then we’ll fix it. - Revisit all Examples to check or validate whether they are fully functional. This is an important review that must be done incrementally. - Retroactively adjust the 10.0.x branch to include Docs, Examples, and Websites. - Provide a new blogging framework for Apache KIE in replacement for https://blog.kie.org/ # The proposal - The Apache KIE website gains a new section called "Docs" in its header, where users are able to select what stream of the Docs they want to see, for example: "Docs" -> "10.1.x" -> "Drools"; or "Docs" -> "10.2.x" -> "SonataFlow" etc. - A new read-only repository called `incubator-kie-examples` is created, containing all examples aligned with development stream branches, and releases candidates and releases tags. - A new release artifact (Examples ZIP) starts existing, and its contents then published to the read-only `incubator-kie-examples` repository. - The existing `incubator-kogito-examples` repository is archived after 10.1 is first released. - A new Continuous Deployment (CD) mechanism is created to ensure Docs and Websites are kept current. # The method Leverage `kie-tools` flexible build system to reduce complexity hidden in Jenkins/GitHub Actions automations, which have proven to become unmaintained and abandoned, by moving content (preserving Git history) from all Docs-, Examples-, and Websites-related repositories into new packages on `kie-tools`. Here’s the final structure this proposal aims to create inside the `kie-tools` repository: - kie-tools/docs/drools - kie-tools/docs/optaplanner - kie-tools/docs/jbpm - kie-tools/docs/kogito - kie-tools/docs/sonataflow - kie-tools/docs/tools - kie-tools/websites/kie-apache-org - kie-tools/examples/rules-* - kie-tools/examples/decisions-* - kie-tools/examples/process-* - kie-tools/examples/sonataflow-* - kie-tools/examples/pmml-* - kie-tools/examples/trusty-* For each example package, Spring Boot, Quarkus, and plain Java subdivisions would be done inside each example using different Maven modules. This would allow users to focus on the domain rather than the framework/implementation and avoid content duplication in READMEs. Being inside `kie-tools`, Examples would be able to benefit from graphical Editors, Runtime Consoles, Quarkus Dev UIs, and Container images, all crucial components for end-to-end solutions built with Apache KIE. This is currently impossible in the current `kogito-examples` repository due to the way our PR checks and releases are structured/automated, where `kogito-examples` builds before `kie-tools`. Making `kogito-examples` build after `kie-tools` for PR checks is non-trivial and completely out-of-scope of this proposal, as it would mean doubling-down on the existing CI systems like “build-chain” for PR checks and “the Kogito framework” on Apache Jenkins, both inherently wasteful on cross-repo PR checks and provenly hard to maintain/operate. Contributions to the proposed new packages would be made following the same commands we already use for all other packages on `kie-tools`. Contributors would need to have `pnpm` and Node.js installed or rely on Devbox to install everything automatically. After setting up the environment, cloning the repo, and running `pnpm bootstrap -F '@kie-docs/drools’ && pnpm -F @kie-docs/drools start` contributors would be able to develop the Drools documentation with live-reload capabilities. The very same would be true for Website(s). For the Examples applications, the process after cloning would be: 1. `pnpm bootstrap -F @kie-examples/some-example-app...` to install 3rd party dependencies and link packages. 2. `pnpm -F @kie-examples/some-example-app^... build:dev` to build this example’s 1st party dependencies. 3. `pnpm -F @kie-examples/some-example-app start` to start the example in its default mode. Each example is also encouraged to define in their README how to start itself using technology-specific commands, like `mvn quarkus:dev -Psome-profile` or even `docker-compose up`. The `start` script is only the default way to run an example, a convention already cohesive with all other existing packages on `kie-tools`. It’s important to note again that the `pnpm` commands would not be necessary in the `incubator-kie-examples` repository and users would be able to run plain `mvn` commands directly inside the example’s directory, for Maven-based examples at least. #### Impact on PR checks (CI) Each new package will contain a `package.json` file describing scripts and their dependency relationship with other packages in the repository. The important scripts are `build:dev`, `build:prod`, and `start`, to integrate them with the existing build system. PRs changing exclusively Docs packages will NOT build anything else in the repository, making PR checks efficient. The same is true for websites. PRs changing example applications will only build their dependency tree and the modified example applications, excluding unrelated packages from the checks while enforcing correctness. This is already how `kie-tools` works. If you need more information on how this mechanism works in practice, please come forward with clarifying questions. We’d love to show you how the partial and partitioned PR checks work on `kie-tools`. #### Impact on Release Process (“CI”) Docs and Websites wouldn’t be release artifacts and would be handled by a new “Continuous deployment (CD)” mechanism. This is covered in the next section of this proposal. With that said, only one new release artifact is being proposed here. A ZIP file containing all examples, stripping out all `kie-tools`-related boilerplate used to develop them. This new artifact (Examples ZIP) would be published to a new read-only repository (incubator-kie-examples) for all release candidates and releases we do. This repository would be THE consumable repository for users who wish to go straight to the point when testing out Apache KIE and who are not necessarily committers/developers. To do that, changing some release automations to add a new job that builds and publishes the Examples ZIP would be necessary: 1. Daily-dev (https://ci-builds.apache.org/job/KIE/job/kie-tools/job/main/job/kie-tools-daily-dev-publish/) 2. Release Candidates (https://ci-builds.apache.org/job/KIE/job/kie-tools/job/main/job/release/job/kie-tools-release-candidate/) 3. Release Promote (https://ci-builds.apache.org/job/KIE/job/kie-tools/job/main/job/release/job/kie-tools-release-promote/ ) #### New Continuous Deployment (CD) Websites and Docs would be treated as continually deployed packages for the development stream we declare as “live”. Also, old development streams would be continually built and published to make documentation updates possible for older versions. Referring to Toshiya's thread on documentation [1], our Docs would be “mutable”, and the granularity would be “per stream”. Two new automations would be created for making sure our content is up-to-date at all times: `publish-kie-apache-org-website`: Responsible for updating live Apache KIE’s website (https://kie.apache.org) and also capable of generating ad-hoc immutable copies of the website for Release Candidates. It would be able to also be executed for the `main` stream for development purposes. `publish-older-docs-to-live-website` ensures that older development streams can have their content updated on the live website. For example, if we find a seriously incorrect statement in an older version, this job would ensure it’s published after being fixed. We can eventually remove this for versions that become too old. # Actionable items (plan) Phase 1 (Development and contributions): - Move all examples to `kie-tools`, already in the new structure/organization, and integrate them as part of the build system - Move all documentation content to `kie-tools` and integrate the new packages as part of the build system - Move `incubator-kie-website` content to `kie-tools` and integrate it as part of the build system - Create new landing pages for each major component: Drools, jBPM, SonataFlow, Optaplanner, and Kogito. Phase 2 (Release Process) - Create the `incubator-kie-examples` repository - Update `kie-tools` daily-dev job to publish the Examples ZIP to `incubator-kie-examples@main` - Update `kie-tools` Release Candidate job to publish the Examples ZIP to https://dist.apache.org/repos/dist/dev/incubator/kie/ - Update `kie-tools` Release Promote job to publish the Examples ZIP content to incubator-kie-examples@{version}` - Archive the `incubator-kie-kogito-examples` repository (post 10.1 release) - Update the Release Procedure for 10.1 to include Docs, Examples, and Websites where necessary Phase 3 (Continuous deployments) - Create and configure new `publish-kie-apache-org-website` job - Create and configure new `publish-docs-to-live-website` job Phase 4 (Cleanup) - Depublish current `drools.org`, `jbpm.org`, `optaplanner.org`, `kie.org`, `sonataflow.org` and all its subdomains. - Redirect all old domains to the proper landing page of Apache KIE’s widely known components (E.g., https://drools.org → https://kie.apache.org/drools) - Preserve Docs from older versions somewhere accessible under Apache KIE’s website. # Commitment and rough timeline Alex Porcelli and Tiago Bento are fully committed to executing this plan in collaboration with other members who want to help. We are committed to working on it as soon as the proposal is approved. Although it’s hard to predict how long it will take, Phases 1, 2, and 3 will take around 6 weeks based on the information we have today. If anything changes or issues are found during execution, we can update this thread. Phase 4 is impossible to estimate, as there might be many different people and/or organizations to chase so that these websites can be decommissioned and properly forward users to KIE’s new home in Apache. # Observations on `kie-tools` This proposal makes Docs, Examples, and Websites available to all users without touching the complex, unmaintained, black-box systems currently in Apache’s Jenkins, known as “build-chain” and “the Kogito CI framework”. We know that `kie-tools` is not unanimously popular among Apache KIE committers, but it’s actively maintained and has been proven to scale well both in package count and technology diversity. We invite everyone to take a look at the repo’s User Manual at https://github.com/apache/incubator-kie-tools/blob/main/repo/MANUAL.md ## Non-obvious positive outcomes - Drools and OptaPlanner repositories won’t have dependencies on JavaScript anymore. - Fonts and other “binary” files won’t need to be committed as a source anymore, as `kie-tools`’s build system can properly depend on NPM artifacts. Also helps towards graduation, as Category X dependencies won’t be present. [1] https://lists.apache.org/thread/zjt156tk5zjw7463xsfdkojwzndwd1mn [2] https://lists.apache.org/thread/qc9vhod96mdoppo5ssj4f0pkqhzt4ghd --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org For additional commands, e-mail: dev-h...@kie.apache.org
