"at the end of the CI chain" On Tue, Feb 4, 2025 at 4:01 PM Francisco Javier Tirado Sarti < ftira...@redhat.com> wrote:
> Jan, > > How costly will it be, keeping the current repo structure, to include > examples of the CI chain? > > On Tue, Feb 4, 2025 at 3:45 PM Jan Šťastný <jstastn...@apache.org> wrote: > >> -0 >> >> What I lack in the proposal is how the move of examples into kie-tools >> resolves the need for running cross-repository PRs or how to solve this >> problem WITHOUT build-chain, which is positioned as a selling point for >> the >> examples move (apart for solving the images-related gaps the pipelines >> have). >> >> If the answer reads that kogito-examples is not meant to test the >> repositories up the chain then I won't agree on that one, albeit current >> shape is only checking compilation rather than testing, but at least that >> (side note we could consider testing the actual readme commands, whole >> different story, e.g. https://github.com/orangemug/readme-tester/ ). >> >> Also there are incorrect assumptions about the reason for build-chain >> existence in the first place - take a look at the repository name >> https://github.com/kiegroup/github-action-build-chain - this is >> primarily a >> GitHub action implemented to enable cross-repository executions (for PRs, >> branch builds, full downstream, etc) in GitHub Actions, jenkins pipelines >> would easily live without the tool, though goal was perhaps to unify the >> means. It also allows local execution in the same way that the CI does, so >> pnpm vs build-chain when it comes to ease of build locally, they seem >> similar to me, except for the single/distributed location of code. >> >> As proposal states: >> >> <cite> >> Making `kogito-examples` build after `kie-tools` for PR >> checks is non-trivial and completely out-of-scope of this proposal, as >> it would mean doubling-down on the existing CI systems like >> “build-chain” for PR checks and “the Kogito framework” on Apache >> Jenkins, both inherently wasteful on cross-repo PR checks and provenly >> hard to maintain/operate. >> </cite> >> >> Which needs more explanation on how we serve the cross-repository PR >> builds >> when a change in kogito-runtimes requires a change in kie-tools examples, >> and that without the build-chain. I guess the response leads to weekly >> SNAPSHOT, which is on its own an artificial construct we invented for >> kie-tools. This way the examples inevitably end as just a project >> depending >> on drools and kogito of a pre-defined version, not a part of the >> environment as such. >> >> kogito-pipelines CI framework assessment does not seem to be strictly tied >> to the given topic, that's a "pipelines as code" framework, which has its >> pros and cons, none of which were actually mentioned as part of the >> proposal. Not entirely sure why it's covered in the text while CI >> refactoring is stated as out of scope. >> >> I am happy to share more on the topic as time allows, though I can't >> commit >> to any counter-proposal in the short term. >> >> Regards >> Jan >> >> On Tue, 4 Feb 2025 at 13:56, Daniel José dos Santos <djdossan...@live.com >> > >> wrote: >> >> > +1 >> > >> > -----Mensagem original----- >> > De: Eder Ignatowicz <ignatow...@gmail.com> >> > Enviada em: segunda-feira, 3 de fevereiro de 2025 13:53 >> > Para: dev@kie.apache.org >> > Assunto: Re: [PROPOSAL][VOTE] Including Docs, Examples, and Website(s) >> in >> > Apache KIE 10.1 and beyond >> > >> > +1 >> > _____________ >> > Eder Ignatowicz >> > ignatow...@gmail.com >> > >> > >> > On Mon, Feb 3, 2025 at 11:52 AM Thiago Lugli <thiago...@apache.org> >> wrote: >> > >> > > +1 >> > > >> > > Em seg., 3 de fev. de 2025 às 13:44, Tiago Bento >> > > <tiagobe...@apache.org> escreveu: >> > > > >> > > > +1 >> > > > >> > > > On Mon, Feb 3, 2025 at 11:39 AM Gabriele Cardosi >> > > > <gabriele.card...@gmail.com> wrote: >> > > > > >> > > > > -1 >> > > > > >> > > > > >> > > > > Il giorno lun 3 feb 2025 alle ore 17:37 Eduardo Cerqueira < >> > > > > eduardocerque...@apache.org> ha scritto: >> > > > > >> > > > > > +1 >> > > > > > >> > > > > > On Mon, Feb 3, 2025 at 11:31 AM Alex Porcelli >> > > > > > <porce...@apache.org> >> > > wrote: >> > > > > > >> > > > > > > Given the recent discussions about Docs [1] and Examples [2], >> > > > > > > Tiago Bento and I got together to write, in proposal form, the >> > > combination >> > > > > > > of our understanding of how we can create the baseline for >> > > > > > > content (Docs, Examples, and Websites) so that our users can >> > > > > > > consume Apache KIE releases in an expected, structured, and >> > > > > > > cohesive way. This proposal will take you < 15 min to read. >> > > > > > > >> > > > > > > - >> > > > > > > Alex & Tiago >> > > > > > > >> > > > > > > # Problem scope >> > > > > > > >> > > > > > > There has been a significant demand for proper guidance on >> > > > > > > using Apache KIE 10.0. Users are unsure how to adopt it, as >> > > > > > > neither Documentation nor Examples were available when 10.0.0 >> > > > > > > was >> > > released. An >> > > > > > > attempt to manually adjust the examples to the 10.0.0 release >> > > > > > > has happened for a couple of weeks now, showing how >> > > > > > > time-consuming it >> > > is >> > > > > > > to defer dealing with “satellite” repositories after releases >> > > > > > > are done. Even more complicated is that well-known websites ( >> > > drools.org, >> > > > > > > jbpm.org, kogito.kie.org) are outdated and out of sync with >> > > > > > > Apache KIE’s repositories and releases, thus contributing to >> > > > > > > more >> > > confusion >> > > > > > > than anything else. >> > > > > > > >> > > > > > > To avoid being unclear about why Docs, Examples, and Websites >> > > > > > > are crucial for the community's success, they are all entry >> > > > > > > points >> > > where >> > > > > > > new users can start understanding Apache KIE, how to use it >> > > > > > > and whether or not it’s a fit for them. For existing users, >> > > > > > > they are >> > > all >> > > > > > > reference material, including code snippets, institutional >> > > > > > > texts and/or vision, licenses, migration guides, and legal >> > material. >> > > > > > > Examples, in particular, also work as playgrounds, where users >> > > > > > > can start with pre-built applications and make minor tweaks to >> > > > > > > either check if something they want is supported or simulate a >> > > > > > > development workflow on a smaller scale. Our code is worth >> > > > > > > close to nothing if there isn’t good supporting material >> around >> > it. >> > > > > > > >> > > > > > > This proposal aims to amend all those problems by bringing >> > > > > > > Docs, Examples, and Websites to the Release Procedure, PR >> > > > > > > checks, and a Continuous deployment (CD) mechanism, making >> > > > > > > them all be part of >> > > our >> > > > > > > daily contributions, so they are not treated as afterthoughts. >> > > > > > > >> > > > > > > >> > > > > > > # What is not in scope >> > > > > > > >> > > > > > > - A complete restructuring of all Apache KIE repositories >> > > > > > > - Refactoring the whole CI; this proposal aims to work with >> > > > > > > the current established CI already in place >> > > > > > > - Refactor content in Docs and Example applications; this >> > > > > > > proposal aims to move stuff as-is, except when something is >> > > > > > > broken, then >> > > we’ll >> > > > > > > fix it. >> > > > > > > - Revisit all Examples to check or validate whether they are >> > > > > > > fully functional. This is an important review that must be >> > > > > > > done incrementally. >> > > > > > > - Retroactively adjust the 10.0.x branch to include Docs, >> > > > > > > Examples, and Websites. >> > > > > > > - Provide a new blogging framework for Apache KIE in >> > > > > > > replacement >> > > for >> > > > > > > https://blog.kie.org/ >> > > > > > > >> > > > > > > >> > > > > > > # The proposal >> > > > > > > >> > > > > > > - The Apache KIE website gains a new section called "Docs" in >> > > > > > > its header, where users are able to select what stream of the >> > > > > > > Docs they want to see, for example: "Docs" -> "10.1.x" -> >> > > > > > > "Drools"; or >> > > "Docs" -> >> > > > > > > "10.2.x" -> "SonataFlow" etc. >> > > > > > > - A new read-only repository called `incubator-kie-examples` >> > > > > > > is created, containing all examples aligned with development >> > > > > > > stream branches, and releases candidates and releases tags. >> > > > > > > - A new release artifact (Examples ZIP) starts existing, and >> > > > > > > its contents then published to the read-only >> > > > > > > `incubator-kie-examples` repository. >> > > > > > > - The existing `incubator-kogito-examples` repository is >> > > > > > > archived after 10.1 is first released. >> > > > > > > - A new Continuous Deployment (CD) mechanism is created to >> > > > > > > ensure >> > > Docs >> > > > > > > and Websites are kept current. >> > > > > > > >> > > > > > > >> > > > > > > # The method >> > > > > > > >> > > > > > > Leverage `kie-tools` flexible build system to reduce >> > > > > > > complexity >> > > hidden >> > > > > > > in Jenkins/GitHub Actions automations, which have proven to >> > > > > > > become unmaintained and abandoned, by moving content >> > > > > > > (preserving Git >> > > history) >> > > > > > > from all Docs-, Examples-, and Websites-related repositories >> > > > > > > into >> > > new >> > > > > > > packages on `kie-tools`. Here’s the final structure this >> > > > > > > proposal >> > > aims >> > > > > > > to create inside the `kie-tools` repository: >> > > > > > > >> > > > > > > - kie-tools/docs/drools >> > > > > > > - kie-tools/docs/optaplanner >> > > > > > > - kie-tools/docs/jbpm >> > > > > > > - kie-tools/docs/kogito >> > > > > > > - kie-tools/docs/sonataflow >> > > > > > > - kie-tools/docs/tools >> > > > > > > - kie-tools/websites/kie-apache-org >> > > > > > > - kie-tools/examples/rules-* >> > > > > > > - kie-tools/examples/decisions-* >> > > > > > > - kie-tools/examples/process-* >> > > > > > > - kie-tools/examples/sonataflow-* >> > > > > > > - kie-tools/examples/pmml-* >> > > > > > > - kie-tools/examples/trusty-* >> > > > > > > >> > > > > > > For each example package, Spring Boot, Quarkus, and plain Java >> > > > > > > subdivisions would be done inside each example using different >> > > Maven >> > > > > > > modules. This would allow users to focus on the domain rather >> > > > > > > than >> > > the >> > > > > > > framework/implementation and avoid content duplication in >> > READMEs. >> > > > > > > >> > > > > > > Being inside `kie-tools`, Examples would be able to benefit >> > > > > > > from graphical Editors, Runtime Consoles, Quarkus Dev UIs, and >> > > > > > > Container images, all crucial components for end-to-end >> > > > > > > solutions built with Apache KIE. This is currently impossible >> > > > > > > in the current `kogito-examples` repository due to the way our >> > > > > > > PR checks and >> > > releases >> > > > > > > are structured/automated, where `kogito-examples` builds >> > > > > > > before `kie-tools`. Making `kogito-examples` build after >> > > > > > > `kie-tools` for >> > > PR >> > > > > > > checks is non-trivial and completely out-of-scope of this >> > > proposal, as >> > > > > > > it would mean doubling-down on the existing CI systems like >> > > > > > > “build-chain” for PR checks and “the Kogito framework” on >> > > > > > > Apache Jenkins, both inherently wasteful on cross-repo PR >> > > > > > > checks and >> > > provenly >> > > > > > > hard to maintain/operate. >> > > > > > > >> > > > > > > Contributions to the proposed new packages would be made >> > > > > > > following >> > > the >> > > > > > > same commands we already use for all other packages on >> > `kie-tools`. >> > > > > > > Contributors would need to have `pnpm` and Node.js installed >> > > > > > > or >> > > rely >> > > > > > > on Devbox to install everything automatically. After setting >> > > > > > > up the environment, cloning the repo, and running `pnpm >> > > > > > > bootstrap -F '@kie-docs/drools’ && pnpm -F @kie-docs/drools >> > > > > > > start` contributors would be able to develop the Drools >> > > > > > > documentation with live-reload capabilities. The very same >> would >> > be true for Website(s). >> > > > > > > >> > > > > > > For the Examples applications, the process after cloning would >> > be: >> > > > > > > >> > > > > > > 1. `pnpm bootstrap -F @kie-examples/some-example-app...` to >> > > > > > > install 3rd party dependencies and link packages. >> > > > > > > 2. `pnpm -F @kie-examples/some-example-app^... build:dev` to >> > > > > > > build this example’s 1st party dependencies. >> > > > > > > 3. `pnpm -F @kie-examples/some-example-app start` to start the >> > > example >> > > > > > > in its default mode. >> > > > > > > >> > > > > > > Each example is also encouraged to define in their README how >> > > > > > > to >> > > start >> > > > > > > itself using technology-specific commands, like `mvn >> > > > > > > quarkus:dev -Psome-profile` or even `docker-compose up`. The >> > > > > > > `start` script is only the default way to run an example, a >> > > > > > > convention already >> > > cohesive >> > > > > > > with all other existing packages on `kie-tools`. >> > > > > > > >> > > > > > > It’s important to note again that the `pnpm` commands would >> > > > > > > not be necessary in the `incubator-kie-examples` repository >> > > > > > > and users >> > > would >> > > > > > > be able to run plain `mvn` commands directly inside the >> > > > > > > example’s directory, for Maven-based examples at least. >> > > > > > > >> > > > > > > >> > > > > > > #### Impact on PR checks (CI) >> > > > > > > >> > > > > > > Each new package will contain a `package.json` file describing >> > > scripts >> > > > > > > and their dependency relationship with other packages in the >> > > > > > > repository. The important scripts are `build:dev`, >> > > > > > > `build:prod`, >> > > and >> > > > > > > `start`, to integrate them with the existing build system. >> > > > > > > >> > > > > > > PRs changing exclusively Docs packages will NOT build anything >> > > else in >> > > > > > > the repository, making PR checks efficient. The same is true >> > > > > > > for websites. >> > > > > > > >> > > > > > > PRs changing example applications will only build their >> > > > > > > dependency tree and the modified example applications, >> > > > > > > excluding unrelated packages from the checks while enforcing >> > > > > > > correctness. This is >> > > already >> > > > > > > how `kie-tools` works. >> > > > > > > >> > > > > > > If you need more information on how this mechanism works in >> > > practice, >> > > > > > > please come forward with clarifying questions. We’d love to >> > > > > > > show >> > > you >> > > > > > > how the partial and partitioned PR checks work on `kie-tools`. >> > > > > > > >> > > > > > > >> > > > > > > #### Impact on Release Process (“CI”) >> > > > > > > >> > > > > > > Docs and Websites wouldn’t be release artifacts and would be >> > > handled >> > > > > > > by a new “Continuous deployment (CD)” mechanism. This is >> > > > > > > covered in the next section of this proposal. >> > > > > > > >> > > > > > > With that said, only one new release artifact is being >> > > > > > > proposed >> > > here. >> > > > > > > A ZIP file containing all examples, stripping out all >> > > > > > > `kie-tools`-related boilerplate used to develop them. This new >> > > > > > > artifact (Examples ZIP) would be published to a new read-only >> > > > > > > repository (incubator-kie-examples) for all release candidates >> > > > > > > and releases we do. This repository would be THE consumable >> > > > > > > repository >> > > for >> > > > > > > users who wish to go straight to the point when testing out >> > > > > > > Apache >> > > KIE >> > > > > > > and who are not necessarily committers/developers. >> > > > > > > >> > > > > > > To do that, changing some release automations to add a new job >> > > > > > > that builds and publishes the Examples ZIP would be necessary: >> > > > > > > >> > > > > > > 1. Daily-dev ( >> > > > > > > >> > > > > > >> > > >> https://ci-builds.apache.org/job/KIE/job/kie-tools/job/main/job/kie-to >> > > ols-daily-dev-publish/ >> > > > > > > ) >> > > > > > > 2. Release Candidates >> > > > > > > ( >> > > > > > > >> > > > > > >> > > >> https://ci-builds.apache.org/job/KIE/job/kie-tools/job/main/job/releas >> > > e/job/kie-tools-release-candidate/ >> > > > > > > ) >> > > > > > > 3. Release Promote >> > > > > > > ( >> > > > > > > >> > > > > > >> > > >> https://ci-builds.apache.org/job/KIE/job/kie-tools/job/main/job/releas >> > > e/job/kie-tools-release-promote/ >> > > > > > > ) >> > > > > > > >> > > > > > > >> > > > > > > #### New Continuous Deployment (CD) >> > > > > > > >> > > > > > > Websites and Docs would be treated as continually deployed >> > > > > > > packages for the development stream we declare as “live”. >> > > > > > > Also, old >> > > development >> > > > > > > streams would be continually built and published to make >> > > documentation >> > > > > > > updates possible for older versions. Referring to Toshiya's >> > > > > > > thread >> > > on >> > > > > > > documentation [1], our Docs would be “mutable”, and the >> > > > > > > granularity would be “per stream”. >> > > > > > > >> > > > > > > Two new automations would be created for making sure our >> > > > > > > content is up-to-date at all times: >> > > > > > > >> > > > > > > `publish-kie-apache-org-website`: Responsible for updating >> > > > > > > live >> > > Apache >> > > > > > > KIE’s website (https://kie.apache.org) and also capable of >> > > generating >> > > > > > > ad-hoc immutable copies of the website for Release Candidates. >> > > > > > > It would be able to also be executed for the `main` stream for >> > > > > > > development purposes. >> > > > > > > >> > > > > > > `publish-older-docs-to-live-website` ensures that older >> > > > > > > development streams can have their content updated on the live >> > > > > > > website. For example, if we find a seriously incorrect >> > > > > > > statement in an older version, this job would ensure it’s >> > > > > > > published after being fixed. We can eventually remove this for >> > versions that become too old. >> > > > > > > >> > > > > > > >> > > > > > > # Actionable items (plan) >> > > > > > > >> > > > > > > Phase 1 (Development and contributions): >> > > > > > > >> > > > > > > - Move all examples to `kie-tools`, already in the new >> > > > > > > structure/organization, and integrate them as part of the >> > > > > > > build >> > > system >> > > > > > > - Move all documentation content to `kie-tools` and >> > > > > > > integrate the new packages as part of the build system >> > > > > > > - Move `incubator-kie-website` content to `kie-tools` and >> > > integrate >> > > > > > > it as part of the build system >> > > > > > > - Create new landing pages for each major component: Drools, >> > > > > > > jBPM, SonataFlow, Optaplanner, and Kogito. >> > > > > > > >> > > > > > > Phase 2 (Release Process) >> > > > > > > >> > > > > > > - Create the `incubator-kie-examples` repository >> > > > > > > - Update `kie-tools` daily-dev job to publish the Examples >> > > > > > > ZIP to `incubator-kie-examples@main` >> > > > > > > - Update `kie-tools` Release Candidate job to publish the >> > > > > > > Examples ZIP to >> > > > > > > https://dist.apache.org/repos/dist/dev/incubator/kie/ >> > > > > > > - Update `kie-tools` Release Promote job to publish the >> > > > > > > Examples >> > > ZIP >> > > > > > > content to incubator-kie-examples@{version}` >> > > > > > > - Archive the `incubator-kie-kogito-examples` repository >> > > > > > > (post >> > > 10.1 >> > > > > > > release) >> > > > > > > - Update the Release Procedure for 10.1 to include Docs, >> > > > > > > Examples, and Websites where necessary >> > > > > > > >> > > > > > > Phase 3 (Continuous deployments) >> > > > > > > >> > > > > > > - Create and configure new `publish-kie-apache-org-website` >> > > > > > > job >> > > > > > > - Create and configure new `publish-docs-to-live-website` job >> > > > > > > >> > > > > > > Phase 4 (Cleanup) >> > > > > > > >> > > > > > > - Depublish current `drools.org`, `jbpm.org`, >> > > > > > > `optaplanner.org`, `kie.org`, `sonataflow.org` and all its >> > subdomains. >> > > > > > > - Redirect all old domains to the proper landing page of >> > > > > > > Apache >> > > KIE’s >> > > > > > > widely known components (E.g., https://drools.org → >> > > > > > > https://kie.apache.org/drools) >> > > > > > > - Preserve Docs from older versions somewhere accessible under >> > > Apache >> > > > > > > KIE’s website. >> > > > > > > >> > > > > > > >> > > > > > > # Commitment and rough timeline >> > > > > > > >> > > > > > > Alex Porcelli and Tiago Bento are fully committed to executing >> > > > > > > this plan in collaboration with other members who want to >> help. >> > > > > > > >> > > > > > > We are committed to working on it as soon as the proposal is >> > > approved. >> > > > > > > Although it’s hard to predict how long it will take, Phases 1, >> > > > > > > 2, >> > > and >> > > > > > > 3 will take around 6 weeks based on the information we have >> > today. >> > > If >> > > > > > > anything changes or issues are found during execution, we can >> > > update >> > > > > > > this thread. >> > > > > > > >> > > > > > > Phase 4 is impossible to estimate, as there might be many >> > > > > > > different people and/or organizations to chase so that these >> > > > > > > websites can be decommissioned and properly forward users to >> > > > > > > KIE’s new home in >> > > Apache. >> > > > > > > >> > > > > > > >> > > > > > > # Observations on `kie-tools` >> > > > > > > >> > > > > > > This proposal makes Docs, Examples, and Websites available to >> > > > > > > all users without touching the complex, unmaintained, >> > > > > > > black-box systems currently in Apache’s Jenkins, known as >> > > > > > > “build-chain” and “the >> > > Kogito >> > > > > > > CI framework”. >> > > > > > > >> > > > > > > We know that `kie-tools` is not unanimously popular among >> > > > > > > Apache >> > > KIE >> > > > > > > committers, but it’s actively maintained and has been proven >> > > > > > > to >> > > scale >> > > > > > > well both in package count and technology diversity. We invite >> > > > > > > everyone to take a look at the repo’s User Manual at >> > > > > > > >> > > >> https://github.com/apache/incubator-kie-tools/blob/main/repo/MANUAL.md >> > > > > > > >> > > > > > > >> > > > > > > ## Non-obvious positive outcomes >> > > > > > > >> > > > > > > - Drools and OptaPlanner repositories won’t have dependencies >> > > > > > > on JavaScript anymore. >> > > > > > > - Fonts and other “binary” files won’t need to be committed as >> > > > > > > a source anymore, as `kie-tools`’s build system can properly >> > > > > > > depend >> > > on >> > > > > > > NPM artifacts. Also helps towards graduation, as Category X >> > > > > > > dependencies won’t be present. >> > > > > > > >> > > > > > > >> > > > > > > [1] >> > > https://lists.apache.org/thread/zjt156tk5zjw7463xsfdkojwzndwd1mn >> > > > > > > [2] >> > > https://lists.apache.org/thread/qc9vhod96mdoppo5ssj4f0pkqhzt4ghd >> > > > > > > >> > > > > > > >> > > --------------------------------------------------------------------- >> > > > > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org For >> > > > > > > additional commands, e-mail: dev-h...@kie.apache.org >> > > > > > > >> > > > > > > >> > > > > > >> > > > >> > > > -------------------------------------------------------------------- >> > > > - To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org For >> > > > additional commands, e-mail: dev-h...@kie.apache.org >> > > > >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org For additional >> > > commands, e-mail: dev-h...@kie.apache.org >> > > >> > > >> > >> >
