Hi everyone, Following up on the status of this effort.
I know I'm a bit late on the email, but it paid off, as we just had our first green build [1] of KIE Tools with Java 17, Quarkus 3.2.9, Maven 3.9.6 and Kogito 999-20240218-SNAPSHOT. We ended up deciding to do items 1. and 2. together after all, but we couldn't use the latest timestamped SNAPSHOT due to https://issues.apache.org/jira/browse/INFRA-25546, which prevented the publication of the timestamped SNAPSHOT to succeed. That's why we picked 20240218, not 20240225. We will proceed with some manual checks now to confirm that everything is working properly, and I hope on the next status update we will have the big blocker removed and that we have already made some progress on items other than 1. and 2.. It's been great seeing everyone working together to move this forward as fast as possible. Thanks a lot Eder, Thiago, Rodrigo, Alex, Fabrizio, Paulo, and Yeser! [1] https://github.com/apache/incubator-kie-tools/actions/runs/8105498131/job/22153897965 On Mon, Feb 26, 2024 at 5:59 AM Francisco Javier Tirado Sarti <[email protected]> wrote: > > Hi Tiago, > It was a nice morning reading ;) Thanks for all the time invested on such > detailed writing. > @Ricardo Zanini <[email protected]> @Walter Medvedeo > <[email protected]> I guess dev ui can be removed from current > examples, at least from the ones I'm familiar with, since their README > files do not really require any graphical tool. But to be honest, since I > have focused my work on SWF simple showcase examples, I'm not 100% sure > that's the same case for more elaborate ones, can you please check that it > is fine to remove dev ui? If this is not the case, maybe those examples can > be moved to the tools repository (points 9 and 10 of Tiago's email) > Thanks in advance. > > On Fri, Feb 23, 2024 at 2:29 AM Tiago Bento <[email protected]> wrote: > > > DISCLAIMER: this is a long email. Feel free to jump to the bottom for > > the concrete task list. > > > > --- > > > > This email aims to explain the issue and lay out the efforts that are > > in place to fix it. > > > > --- > > > > Before we start digging into the problem, we need to understand a > > little bit of the history of the KIE community and the Kogito > > initiative, at least from the part I started contributing to it anyway > > :) > > > > Back in Business Central days, we had a single stream of development. > > We had robust CI jobs that would build the entire stack, from `drools` > > to `kie-wb-distributions`. When Kogito came about, there was a fission > > in our pipelines. Kogito started to release independently from > > everything else, and because of the organic way things started to > > organize themselves, the Kogito Tooling initiative needed to have its > > own release cycle as well. > > > > Up until we started the migration to Apache, Business Central > > continued on its `7.x` stream, Kogito was at `1.x` stream, and Kogito > > Tooling (which was already called KIE Tools) was still on `0.x`. > > Drools started its own release stream as well, with `8.x`, and > > OptaPlanner I believe had jumped to `9.x` already. This fragmentation > > had technical implications, which we are trying to overcome right now > > with Apache KIE 10. > > > > The Apache KIE 10 release is aimed to be an atomic release, where the > > entire Apache KIE community follows a single release stream. This > > means we need to make sure our code base is in sync, from `drools` to > > `kie-tools`. > > > > This whole conversation of course doesn't come without a background of > > CI pipelines, organizational perspectives of what Apache KIE is, and > > even poly- vs. mono-repo discussions. This is not the focus of this > > issue, and I believe with time we'll converge into something everyone > > is comfortable with and that ultimately serves our users well. > > > > --- > > > > Due to the lack of unification and flexibility while bootstrapping new > > efforts, KIE Tools had to come up with a way to depend on the new > > stuff (i.e. Kogito) without sacrificing its own integrity and > > stability. KIE Tools, as many of you know, evolved to be a polyglot > > monorepo, hosting Java, JavaScript/TypeScript, Go and Docker images. > > There's a multitude of technologies at play on KIE Tools, which > > ultimately gives our users KIE Sandbox, VS Code and Chrome Extensions, > > and Standalone Editors that can be used as regular libraries on web > > applications. > > > > The technical solution we found on `kie-tools` was depending on Kogito > > as if it were a 3rd party dependency. We have a single, fixed version > > of it and we manually upgrade it to the latest one available when we > > need to. This has worked amazingly well, allowing us to own the > > codebase and avoid random breaks coming from SNAPSHOT artifacts. > > > > This allowed the `drools` and `kogito-*` repositories to evolve > > independently of `kie-tools`. This also made `kie-tools` responsible > > for adapting itself to changes coming from `drools` and `kogito-*`. > > Luckily, we didn't have many incompatibilities, but right now we have > > an important one, that we'll discuss below. > > > > Note that at this point, and for this issue in particular, it is > > unimportant to discuss why things were done the way they were, or what > > we should've done instead. I'm laying out the facts so that we can > > understand how we're moving forward now. Of course, I would personally > > love to debate this on a separate thread :) > > > > --- > > > > One of the most impactful features of KIE Sandbox is the DMN Runner. > > It allows people to rapidly interact with their Decisions as they > > author them, and this was all made possible because of the JIT > > Executor (hosted at `kogito-apps`). KIE Sandbox also started to allow > > users to deploy their Decisions to OpenShift, then later to > > Kubernetes, and now recently we allow users to deploy anything to > > OpenShift and Kubernetes, with a very high level of customization. All > > these features depend on `kogito-*`, Quarkus, and all the things that > > come with them. Including the Java version. > > > > Logically, `kie-tools` depends on `drools` and `kogito-*`, as `drools` > > and `kogito-*` can (and do!) exist and function without Tools, but not > > the other way around. > > > > As `kie-tools` started growing, maturing, and releasing some pieces of > > technology that were useful to other domains (i.e. Multiplying > > Architecture), it became a dependency for other projects. Users > > started depending on our Standalone Editors, but also on the > > Multiplying Architecture packages. One good example is Kaoto, which > > released a VS Code extension aligned with our architecture. > > > > The capability of encapsulating web components in any technology > > behind a well defined, statically typed API was really appealing, and > > when the Kogito initiative started to design their Runtime Tools > > (i.e., Quarkus Dev UIs, Management Console, and Task Inbox), using the > > Multiplying Architecture was the way to go. > > > > That's when the circular dependency started. We had > > `kogito-apps/ui-packages` depending on `kie-tools`. Note that it was > > also logical for these packages to be part of the `kogito-*` > > repositories and release stream. Users upgrading to a new version of > > Kogito would also need to have their Task Inbox, Management Console, > > and Quarkus Dev UIs aligned with the new version of Kogito. Not to > > mention Job Service, Data Index etc. > > > > For many months (or years!), this was not an issue, as the Multiplying > > Architecture packages were very stable and very rarely were modified. > > The last big change we had was 3 years ago, with the introduction of > > Shared Values [1], which did not impact the fire-and-forget and > > Promise-based mechanisms we already had in place. > > > > Although we were aware of this circular dependency from the beginning, > > it was not that big of a deal for `kogito-apps` to depend on an older, > > dephased version of `kie-tools`. We didn't have plans of unifying the > > streams anyway, and it was the best solution for our users to have > > their Runtime Tools packages released together with the Kogito stream. > > > > That of course changed after the migration to Apache. > > > > --- > > > > If you reached this point, thank you for your interest in a little bit > > of the history of KIE, and I hope you can now understand why things > > are the way they are. Let's talk about what changed and what we are > > doing to adapt to the new requirements. > > > > Hinting a little bit to what I mentioned above, this discussion is > > very intertwined with the CI pipelines and versioning/releasing > > strategies subjects, as we are essentially discussing code > > organization, coupling/decoupling and granularity of stuff, and to > > make things easier at least for a while, `drools` and `kogito-*` will > > continue being a separate stream from `kie-tools`. > > > > `drools` and `kogito-*` will continue being able to evolve > > independently of `kie-tools`, and `kie-tools` will continue adapting > > itself to whatever changes on `drools` and `kogito-*`. > > > > At the same time, Apache KIE 10 (and the next releases too!) will be > > atomic. And for our users, a single release stream. `drools@10` and > > `kogito-*@10` will be used on `kie-tools@10`. We started this effort a > > while ago, and although we should've been more open about it from the > > beginning, we are also learning how to properly communicate inside the > > Apache structure. Better late than never :) Thanks for the tip Tibor! > > > > Here's how we'll do it: > > > > 1. A couple of weeks ago, one of our newest committers. Rodrigo > > Antunes, created automated jobs to weekly publish a timestamped > > SNAPSHOT version of `drools` and `kogito-*`. This will allow us to > > have a persistent, immutable version to point to on `kie-tools`, even > > without having a full release. Timestamped SNAPSHOTs, as we're calling > > them, are a little bit different than normal Maven SNAPSHOTs, as there > > is no `updatePolicy`, so as time passes, the downloaded artifacts do > > not change. This allows `kie-tools` to be stable and incorporate > > changes done to `drools` and `kogito-*` at its own pace. We also moved > > the publication of `@kie-tools/jitexecutor-native` to this new job, so > > we won't have to maintain it on the `kie-tools` side anymore. Thanks > > Rodrigo! > > > > 2. When we define the code freeze prior to releasing Apache KIE, > > `drools` and `kogito-*` will need to be frozen a little bit earlier > > than `kie-tools`, leaving a buffer for `kie-tools` to start pointing > > to the latest timestamped SNAPSHOT and adapting itself to the changes. > > My initial thoughts are around 2 weeks, but with constant updates and > > proper alignment between the two streams, I see this time reducing to > > a couple of days for upcoming releases. > > > > 3. After `kie-tools` is fully functional pointing to the latest > > timestamped SNAPSHOT of `drools` and `kogito-*`, we can now start > > preparing for the actual release. We will cut new branches for the > > next version (i.e. 10.0.0) and update all configuration files that we > > have to. This will be done on `drools`, `kogito-*`, and `kie-tools`. > > Every module/package will declare the same version, and will depend on > > the upstream modules/packages with the same version as well. We can > > proceed with this staging phase as normal. Running automated/manual > > tests and eventually signing off that everything is good to go. (I'm > > not 100% on the timeline for the Apache release, but with this process > > we can guarantee that we will vote on branches that won't change when > > we decide to release. It will be a matter of pushing tags pointing to > > these release branches). > > > > With the release process defined, we can now discuss the exact actions > > we're taking to remove the circular dependency we have between > > `kie-tools` and `kogito-apps`. All of which will need to be addressed > > prior to starting the release process. > > > > 1. (Fabrizio - IN PROGRESS) Make sure `kie-tools` is on Java 17, Maven > > 3.9.6, and Quarkus 3. > > (https://github.com/apache/incubator-kie-issues/issues/960) > > There's a complication here, since we still have three big > > packages (Serverless Workflow Diagram Editor, BPMN, DMN, and SceSim > > Editors, and DashBuilder) on `kie-tools` that need GWT, and as you > > know, GWT together with Java upgrades usually require some work. We're > > on GWT 2.10, which should already support Java 17, but we'll know more > > once we try it. > > > > 2. (Fabrizio - NOT YET STARTED) Make sure `kie-tools` is using the > > latest timestamped SNAPSHOT from `kogito-*`. > > (https://github.com/apache/incubator-kie-issues/issues/965) > > We have them built and published every Sunday EOD, so on Monday > > (Feb 25th, 2024) we'll have a brand new one to start working with. > > > > 3. (Thiago - IN PROGRESS) Move `kogito-apps/ui-packages` to > > `kie-tools`. (https://github.com/apache/incubator-kie-issues/issues/833) > > We started a couple of weeks ago, and we should be ~80% done with it > > by now. > > > > 4. (Fabrizio - PR SENT, but blocked by 1. and 2.) Move the SWF Quarkus > > Dev UI package from `kogito-apps` to `kie-tools`. > > (https://github.com/apache/incubator-kie-tools/pull/2167) > > This can only be done after 1. and 2. are merged, since this move > > will require `kie-tools` to be ready to build packages with Java 17, > > Maven 3.9.6, and Quarkus 3. > > > > 5. (Thiago - NOT YET STARTED) Move the BPMN Quarkus Dev UI package > > from `kogito-apps` to `kie-tools`. > > (https://github.com/apache/incubator-kie-issues/issues/966) > > We'll follow the lead of Fabrizio's PR from 4. That's essentially > > doing exactly the same thing for the BPMN Quarkus Dev UI module. > > > > 6. (Rodrigo - NOT YET STARTED) Change the Jenkins release jobs to > > include publishing both Quarkus Dev UI modules to Maven central as > > part of the Apache release. > > (https://github.com/apache/incubator-kie-issues/issues/964) > > Should be pretty straight-forward, as we already have experience > > doing this for other repos. > > > > 7. (Thiago - NOT YET STARTED) Change the ownership of `task-console` > > and `management-console` images from `kogito-images` to `kie-tools`. > > (https://github.com/apache/incubator-kie-issues/issues/963) > > KIE Tools already hosts and publishes several images, and since > > the content of those images will now be hosted at `kie-tools`, we can > > only host them there. > > This also means we'll get rid of the two Quarkus apps [2] that > > wrap both consoles' static assets. We could be wrong about the purpose > > of those apps, so if anyone has more information, please do reach out. > > Otherwise, we'll skip the Quarkusification of those static assets and > > we'll ship images with `httpd` directly, like we do for KIE Sandbox. > > > > 8. (Assignee TDB - NOT YET STARTED) Deleting everything that was moved > > to `kie-tools` from `kogito-apps`. > > (https://github.com/apache/incubator-kie-issues/issues/962) > > That will be the point where we actually get rid of the circular > > dependency and remove the blocker for the release. > > > > 9. (Assignee TDB - NOT YET STARTED) Adapt `kogito-examples` to all the > > changes we mentioned above. > > (https://github.com/apache/incubator-kie-issues/issues/961) > > Kogito Examples keeps being part of the `drools` and `kogito-*` > > stream, meaning that example modules won't be able to use neither of > > the Quarkus Dev UIs, as `kie-tools` is downstream in relation to the > > `drools` and `kogito-*` stream. > > The same would be true for Management Console and Task Console > > images, but for them, we can use the `daily-dev` tag we publish daily > > from `kie-tools@main`. > > > > 10. (Assignee TDB - NOT YET STARTED) Create some example apps that use > > the Quarkus Dev UIs directly on the `examples` directory of > > `kie-tools`. (https://github.com/apache/incubator-kie-issues/issues/959) > > This is to serve the same purpose of packages that did that on > > `kogito-examples`, but now hosted on `kie-tools`. > > > > Pheew! That's a lot of things to do, but it is the result of two > > intense weeks assessing and planning how we would address this issue > > in a way that the Apache KIE 10 release can be done in a way that > > feels atomic to our users, who are ultimately the most important part > > of the community. There's an open EPIC too at > > https://github.com/apache/incubator-kie-issues/issues/967. > > > > I appreciate it if you reached this point, and hope you feel welcome > > to contributing to solving this lengthy and historical problem if you > > feel like doing so. > > > > I'll personally send updates to this thread every week, on Thursday > > nights. Do not hesitate to contact me or any of the people involved in > > this effort if you have questions, or simply want to chat about it. > > > > Thanks a lot Alex, Paulo, Fabrizio, Thiago, Rodrigo, and others that > > somehow helped so far. > > > > Kind regards, > > > > Tiago Bento > > > > --- > > > > P.S.: You might've noticed we left Trusty out of the picture for now. > > Thiago is conducting an investigation to understand what would be the > > best course of action for it. As much as we have plans to revive it, > > right now our focus has to be on getting Apache KIE 10 out. We can > > always rely on Git to bring it back in the future, if we ultimately > > decide that wiping out Trusty is the best strategy for this moment. > > > > [1] https://github.com/apache/incubator-kie-tools/pull/691 > > > > [2] > > https://github.com/apache/incubator-kie-kogito-apps/tree/main/task-console > > and > > https://github.com/apache/incubator-kie-kogito-apps/tree/main/management-console > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
