DISCLAIMER: this is a long email. Feel free to jump to the bottom for the concrete task list.
--- This email aims to explain the issue and lay out the efforts that are in place to fix it. --- Before we start digging into the problem, we need to understand a little bit of the history of the KIE community and the Kogito initiative, at least from the part I started contributing to it anyway :) Back in Business Central days, we had a single stream of development. We had robust CI jobs that would build the entire stack, from `drools` to `kie-wb-distributions`. When Kogito came about, there was a fission in our pipelines. Kogito started to release independently from everything else, and because of the organic way things started to organize themselves, the Kogito Tooling initiative needed to have its own release cycle as well. Up until we started the migration to Apache, Business Central continued on its `7.x` stream, Kogito was at `1.x` stream, and Kogito Tooling (which was already called KIE Tools) was still on `0.x`. Drools started its own release stream as well, with `8.x`, and OptaPlanner I believe had jumped to `9.x` already. This fragmentation had technical implications, which we are trying to overcome right now with Apache KIE 10. The Apache KIE 10 release is aimed to be an atomic release, where the entire Apache KIE community follows a single release stream. This means we need to make sure our code base is in sync, from `drools` to `kie-tools`. This whole conversation of course doesn't come without a background of CI pipelines, organizational perspectives of what Apache KIE is, and even poly- vs. mono-repo discussions. This is not the focus of this issue, and I believe with time we'll converge into something everyone is comfortable with and that ultimately serves our users well. --- Due to the lack of unification and flexibility while bootstrapping new efforts, KIE Tools had to come up with a way to depend on the new stuff (i.e. Kogito) without sacrificing its own integrity and stability. KIE Tools, as many of you know, evolved to be a polyglot monorepo, hosting Java, JavaScript/TypeScript, Go and Docker images. There's a multitude of technologies at play on KIE Tools, which ultimately gives our users KIE Sandbox, VS Code and Chrome Extensions, and Standalone Editors that can be used as regular libraries on web applications. The technical solution we found on `kie-tools` was depending on Kogito as if it were a 3rd party dependency. We have a single, fixed version of it and we manually upgrade it to the latest one available when we need to. This has worked amazingly well, allowing us to own the codebase and avoid random breaks coming from SNAPSHOT artifacts. This allowed the `drools` and `kogito-*` repositories to evolve independently of `kie-tools`. This also made `kie-tools` responsible for adapting itself to changes coming from `drools` and `kogito-*`. Luckily, we didn't have many incompatibilities, but right now we have an important one, that we'll discuss below. Note that at this point, and for this issue in particular, it is unimportant to discuss why things were done the way they were, or what we should've done instead. I'm laying out the facts so that we can understand how we're moving forward now. Of course, I would personally love to debate this on a separate thread :) --- One of the most impactful features of KIE Sandbox is the DMN Runner. It allows people to rapidly interact with their Decisions as they author them, and this was all made possible because of the JIT Executor (hosted at `kogito-apps`). KIE Sandbox also started to allow users to deploy their Decisions to OpenShift, then later to Kubernetes, and now recently we allow users to deploy anything to OpenShift and Kubernetes, with a very high level of customization. All these features depend on `kogito-*`, Quarkus, and all the things that come with them. Including the Java version. Logically, `kie-tools` depends on `drools` and `kogito-*`, as `drools` and `kogito-*` can (and do!) exist and function without Tools, but not the other way around. As `kie-tools` started growing, maturing, and releasing some pieces of technology that were useful to other domains (i.e. Multiplying Architecture), it became a dependency for other projects. Users started depending on our Standalone Editors, but also on the Multiplying Architecture packages. One good example is Kaoto, which released a VS Code extension aligned with our architecture. The capability of encapsulating web components in any technology behind a well defined, statically typed API was really appealing, and when the Kogito initiative started to design their Runtime Tools (i.e., Quarkus Dev UIs, Management Console, and Task Inbox), using the Multiplying Architecture was the way to go. That's when the circular dependency started. We had `kogito-apps/ui-packages` depending on `kie-tools`. Note that it was also logical for these packages to be part of the `kogito-*` repositories and release stream. Users upgrading to a new version of Kogito would also need to have their Task Inbox, Management Console, and Quarkus Dev UIs aligned with the new version of Kogito. Not to mention Job Service, Data Index etc. For many months (or years!), this was not an issue, as the Multiplying Architecture packages were very stable and very rarely were modified. The last big change we had was 3 years ago, with the introduction of Shared Values [1], which did not impact the fire-and-forget and Promise-based mechanisms we already had in place. Although we were aware of this circular dependency from the beginning, it was not that big of a deal for `kogito-apps` to depend on an older, dephased version of `kie-tools`. We didn't have plans of unifying the streams anyway, and it was the best solution for our users to have their Runtime Tools packages released together with the Kogito stream. That of course changed after the migration to Apache. --- If you reached this point, thank you for your interest in a little bit of the history of KIE, and I hope you can now understand why things are the way they are. Let's talk about what changed and what we are doing to adapt to the new requirements. Hinting a little bit to what I mentioned above, this discussion is very intertwined with the CI pipelines and versioning/releasing strategies subjects, as we are essentially discussing code organization, coupling/decoupling and granularity of stuff, and to make things easier at least for a while, `drools` and `kogito-*` will continue being a separate stream from `kie-tools`. `drools` and `kogito-*` will continue being able to evolve independently of `kie-tools`, and `kie-tools` will continue adapting itself to whatever changes on `drools` and `kogito-*`. At the same time, Apache KIE 10 (and the next releases too!) will be atomic. And for our users, a single release stream. `drools@10` and `kogito-*@10` will be used on `kie-tools@10`. We started this effort a while ago, and although we should've been more open about it from the beginning, we are also learning how to properly communicate inside the Apache structure. Better late than never :) Thanks for the tip Tibor! Here's how we'll do it: 1. A couple of weeks ago, one of our newest committers. Rodrigo Antunes, created automated jobs to weekly publish a timestamped SNAPSHOT version of `drools` and `kogito-*`. This will allow us to have a persistent, immutable version to point to on `kie-tools`, even without having a full release. Timestamped SNAPSHOTs, as we're calling them, are a little bit different than normal Maven SNAPSHOTs, as there is no `updatePolicy`, so as time passes, the downloaded artifacts do not change. This allows `kie-tools` to be stable and incorporate changes done to `drools` and `kogito-*` at its own pace. We also moved the publication of `@kie-tools/jitexecutor-native` to this new job, so we won't have to maintain it on the `kie-tools` side anymore. Thanks Rodrigo! 2. When we define the code freeze prior to releasing Apache KIE, `drools` and `kogito-*` will need to be frozen a little bit earlier than `kie-tools`, leaving a buffer for `kie-tools` to start pointing to the latest timestamped SNAPSHOT and adapting itself to the changes. My initial thoughts are around 2 weeks, but with constant updates and proper alignment between the two streams, I see this time reducing to a couple of days for upcoming releases. 3. After `kie-tools` is fully functional pointing to the latest timestamped SNAPSHOT of `drools` and `kogito-*`, we can now start preparing for the actual release. We will cut new branches for the next version (i.e. 10.0.0) and update all configuration files that we have to. This will be done on `drools`, `kogito-*`, and `kie-tools`. Every module/package will declare the same version, and will depend on the upstream modules/packages with the same version as well. We can proceed with this staging phase as normal. Running automated/manual tests and eventually signing off that everything is good to go. (I'm not 100% on the timeline for the Apache release, but with this process we can guarantee that we will vote on branches that won't change when we decide to release. It will be a matter of pushing tags pointing to these release branches). With the release process defined, we can now discuss the exact actions we're taking to remove the circular dependency we have between `kie-tools` and `kogito-apps`. All of which will need to be addressed prior to starting the release process. 1. (Fabrizio - IN PROGRESS) Make sure `kie-tools` is on Java 17, Maven 3.9.6, and Quarkus 3. (https://github.com/apache/incubator-kie-issues/issues/960) There's a complication here, since we still have three big packages (Serverless Workflow Diagram Editor, BPMN, DMN, and SceSim Editors, and DashBuilder) on `kie-tools` that need GWT, and as you know, GWT together with Java upgrades usually require some work. We're on GWT 2.10, which should already support Java 17, but we'll know more once we try it. 2. (Fabrizio - NOT YET STARTED) Make sure `kie-tools` is using the latest timestamped SNAPSHOT from `kogito-*`. (https://github.com/apache/incubator-kie-issues/issues/965) We have them built and published every Sunday EOD, so on Monday (Feb 25th, 2024) we'll have a brand new one to start working with. 3. (Thiago - IN PROGRESS) Move `kogito-apps/ui-packages` to `kie-tools`. (https://github.com/apache/incubator-kie-issues/issues/833) We started a couple of weeks ago, and we should be ~80% done with it by now. 4. (Fabrizio - PR SENT, but blocked by 1. and 2.) Move the SWF Quarkus Dev UI package from `kogito-apps` to `kie-tools`. (https://github.com/apache/incubator-kie-tools/pull/2167) This can only be done after 1. and 2. are merged, since this move will require `kie-tools` to be ready to build packages with Java 17, Maven 3.9.6, and Quarkus 3. 5. (Thiago - NOT YET STARTED) Move the BPMN Quarkus Dev UI package from `kogito-apps` to `kie-tools`. (https://github.com/apache/incubator-kie-issues/issues/966) We'll follow the lead of Fabrizio's PR from 4. That's essentially doing exactly the same thing for the BPMN Quarkus Dev UI module. 6. (Rodrigo - NOT YET STARTED) Change the Jenkins release jobs to include publishing both Quarkus Dev UI modules to Maven central as part of the Apache release. (https://github.com/apache/incubator-kie-issues/issues/964) Should be pretty straight-forward, as we already have experience doing this for other repos. 7. (Thiago - NOT YET STARTED) Change the ownership of `task-console` and `management-console` images from `kogito-images` to `kie-tools`. (https://github.com/apache/incubator-kie-issues/issues/963) KIE Tools already hosts and publishes several images, and since the content of those images will now be hosted at `kie-tools`, we can only host them there. This also means we'll get rid of the two Quarkus apps [2] that wrap both consoles' static assets. We could be wrong about the purpose of those apps, so if anyone has more information, please do reach out. Otherwise, we'll skip the Quarkusification of those static assets and we'll ship images with `httpd` directly, like we do for KIE Sandbox. 8. (Assignee TDB - NOT YET STARTED) Deleting everything that was moved to `kie-tools` from `kogito-apps`. (https://github.com/apache/incubator-kie-issues/issues/962) That will be the point where we actually get rid of the circular dependency and remove the blocker for the release. 9. (Assignee TDB - NOT YET STARTED) Adapt `kogito-examples` to all the changes we mentioned above. (https://github.com/apache/incubator-kie-issues/issues/961) Kogito Examples keeps being part of the `drools` and `kogito-*` stream, meaning that example modules won't be able to use neither of the Quarkus Dev UIs, as `kie-tools` is downstream in relation to the `drools` and `kogito-*` stream. The same would be true for Management Console and Task Console images, but for them, we can use the `daily-dev` tag we publish daily from `kie-tools@main`. 10. (Assignee TDB - NOT YET STARTED) Create some example apps that use the Quarkus Dev UIs directly on the `examples` directory of `kie-tools`. (https://github.com/apache/incubator-kie-issues/issues/959) This is to serve the same purpose of packages that did that on `kogito-examples`, but now hosted on `kie-tools`. Pheew! That's a lot of things to do, but it is the result of two intense weeks assessing and planning how we would address this issue in a way that the Apache KIE 10 release can be done in a way that feels atomic to our users, who are ultimately the most important part of the community. There's an open EPIC too at https://github.com/apache/incubator-kie-issues/issues/967. I appreciate it if you reached this point, and hope you feel welcome to contributing to solving this lengthy and historical problem if you feel like doing so. I'll personally send updates to this thread every week, on Thursday nights. Do not hesitate to contact me or any of the people involved in this effort if you have questions, or simply want to chat about it. Thanks a lot Alex, Paulo, Fabrizio, Thiago, Rodrigo, and others that somehow helped so far. Kind regards, Tiago Bento --- P.S.: You might've noticed we left Trusty out of the picture for now. Thiago is conducting an investigation to understand what would be the best course of action for it. As much as we have plans to revive it, right now our focus has to be on getting Apache KIE 10 out. We can always rely on Git to bring it back in the future, if we ultimately decide that wiping out Trusty is the best strategy for this moment. [1] https://github.com/apache/incubator-kie-tools/pull/691 [2] https://github.com/apache/incubator-kie-kogito-apps/tree/main/task-console and https://github.com/apache/incubator-kie-kogito-apps/tree/main/management-console --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
