Hi Tiago, that is a problem, we are removing an important functionality for our SWF users, imho it is a killer feature in this case. I am against it, there needs to be a solution where we can depend on kie-tools with kie-kogito-images.
Another thing that is currently open as a PR [2] that makes images depend on kie-tools. Here we are following the approach you started in kiegroup and we were not aware of any changes, now with the unexpected and sudden changes it is irrelevant? Do you have a proposal on how to achieve what we have in [2] after the circular dependency issue is fixed according to the proposal annoucement? Dominik [2] https://github.com/apache/incubator-kie-kogito-images/pull/1734 On Wed, 6 Mar 2024 at 02:10, Tiago Bento <[email protected]> wrote: > Hi Dominik, > > That is indeed an oversight on our part. Thanks for pointing that out. > > Unfortunately, given the two development streams (`drools` and > `kogito-*`, and `kie-tools`), we can't have `kogito-images` depending > on packages that will start being hosted on `kie-tools`, since > `kogito-images` is part of the `drools` and `kogito-*` development > stream, which is upstream from `kie-tools`' perspective. > > Since all Dev UI packages are moving to `kie-tools`, images on > `kogito-images` can't depend on them. > > I'm not familiar with the structure of `kogito-images`, nor the > purpose of this image that uses SWF's Quarkus Dev UI, but following > the same reasoning we did for `kogito-management-console` and > `kogito-task-console`, my suggestion would be moving this image to > `kie-tools`. Another option I can see is simply removing the Quarkus > Dev UI dependency on this image. > > I'd wait for other opinions, though, as I'm not the best person to > answer SWF-related questions. > > Regards, > > Tiago Bento > > > On Tue, Mar 5, 2024 at 9:31 AM Dominik Hanak <[email protected]> > wrote: > > > > Hi Tiago, > > > > Thanks for moving this forward. > > > > I have one thing to highlight with > > https://github.com/apache/incubator-kie-kogito-images > > I noticed [1], where the swf devui is being used in an image. Imho that > > would require adjustments to the nightly pipelines & release process. > > Would it be possible to add tasks to the plan that ensure required > > adjustments to our CI once the migration is done? > > > > Best regards, > > > > Dominik > > > > [1] > > > https://github.com/apache/incubator-kie-kogito-images/blob/ce62b9d86c2ea0385ab85c3ef5339afe5ecdea72/modules/kogito-swf/devmode/build-config/module.yaml#L30 > > > > > > On Fri, 1 Mar 2024 at 05:54, Tiago Bento <[email protected]> wrote: > > > > > Hi everyone, > > > > > > Following up on the status of this effort. > > > > > > I know I'm a bit late on the email, but it paid off, as we just had > > > our first green build [1] of KIE Tools with Java 17, Quarkus 3.2.9, > > > Maven 3.9.6 and Kogito 999-20240218-SNAPSHOT. We ended up deciding to > > > do items 1. and 2. together after all, but we couldn't use the latest > > > timestamped SNAPSHOT due to > > > https://issues.apache.org/jira/browse/INFRA-25546, which prevented the > > > publication of the timestamped SNAPSHOT to succeed. That's why we > > > picked 20240218, not 20240225. > > > > > > We will proceed with some manual checks now to confirm that everything > > > is working properly, and I hope on the next status update we will have > > > the big blocker removed and that we have already made some progress on > > > items other than 1. and 2.. > > > > > > It's been great seeing everyone working together to move this forward > > > as fast as possible. Thanks a lot Eder, Thiago, Rodrigo, Alex, > > > Fabrizio, Paulo, and Yeser! > > > > > > [1] > > > > https://github.com/apache/incubator-kie-tools/actions/runs/8105498131/job/22153897965 > > > > > > On Mon, Feb 26, 2024 at 5:59 AM Francisco Javier Tirado Sarti > > > <[email protected]> wrote: > > > > > > > > Hi Tiago, > > > > It was a nice morning reading ;) Thanks for all the time invested on > such > > > > detailed writing. > > > > @Ricardo Zanini <[email protected]> @Walter Medvedeo > > > > <[email protected]> I guess dev ui can be removed from > current > > > > examples, at least from the ones I'm familiar with, since their > README > > > > files do not really require any graphical tool. But to be honest, > since I > > > > have focused my work on SWF simple showcase examples, I'm not 100% > sure > > > > that's the same case for more elaborate ones, can you please check > that > > > it > > > > is fine to remove dev ui? If this is not the case, maybe those > examples > > > can > > > > be moved to the tools repository (points 9 and 10 of Tiago's email) > > > > Thanks in advance. > > > > > > > > On Fri, Feb 23, 2024 at 2:29 AM Tiago Bento <[email protected]> > > > wrote: > > > > > > > > > DISCLAIMER: this is a long email. Feel free to jump to the bottom > for > > > > > the concrete task list. > > > > > > > > > > --- > > > > > > > > > > This email aims to explain the issue and lay out the efforts that > are > > > > > in place to fix it. > > > > > > > > > > --- > > > > > > > > > > Before we start digging into the problem, we need to understand a > > > > > little bit of the history of the KIE community and the Kogito > > > > > initiative, at least from the part I started contributing to it > anyway > > > > > :) > > > > > > > > > > Back in Business Central days, we had a single stream of > development. > > > > > We had robust CI jobs that would build the entire stack, from > `drools` > > > > > to `kie-wb-distributions`. When Kogito came about, there was a > fission > > > > > in our pipelines. Kogito started to release independently from > > > > > everything else, and because of the organic way things started to > > > > > organize themselves, the Kogito Tooling initiative needed to have > its > > > > > own release cycle as well. > > > > > > > > > > Up until we started the migration to Apache, Business Central > > > > > continued on its `7.x` stream, Kogito was at `1.x` stream, and > Kogito > > > > > Tooling (which was already called KIE Tools) was still on `0.x`. > > > > > Drools started its own release stream as well, with `8.x`, and > > > > > OptaPlanner I believe had jumped to `9.x` already. This > fragmentation > > > > > had technical implications, which we are trying to overcome right > now > > > > > with Apache KIE 10. > > > > > > > > > > The Apache KIE 10 release is aimed to be an atomic release, where > the > > > > > entire Apache KIE community follows a single release stream. This > > > > > means we need to make sure our code base is in sync, from `drools` > to > > > > > `kie-tools`. > > > > > > > > > > This whole conversation of course doesn't come without a > background of > > > > > CI pipelines, organizational perspectives of what Apache KIE is, > and > > > > > even poly- vs. mono-repo discussions. This is not the focus of this > > > > > issue, and I believe with time we'll converge into something > everyone > > > > > is comfortable with and that ultimately serves our users well. > > > > > > > > > > --- > > > > > > > > > > Due to the lack of unification and flexibility while bootstrapping > new > > > > > efforts, KIE Tools had to come up with a way to depend on the new > > > > > stuff (i.e. Kogito) without sacrificing its own integrity and > > > > > stability. KIE Tools, as many of you know, evolved to be a polyglot > > > > > monorepo, hosting Java, JavaScript/TypeScript, Go and Docker > images. > > > > > There's a multitude of technologies at play on KIE Tools, which > > > > > ultimately gives our users KIE Sandbox, VS Code and Chrome > Extensions, > > > > > and Standalone Editors that can be used as regular libraries on web > > > > > applications. > > > > > > > > > > The technical solution we found on `kie-tools` was depending on > Kogito > > > > > as if it were a 3rd party dependency. We have a single, fixed > version > > > > > of it and we manually upgrade it to the latest one available when > we > > > > > need to. This has worked amazingly well, allowing us to own the > > > > > codebase and avoid random breaks coming from SNAPSHOT artifacts. > > > > > > > > > > This allowed the `drools` and `kogito-*` repositories to evolve > > > > > independently of `kie-tools`. This also made `kie-tools` > responsible > > > > > for adapting itself to changes coming from `drools` and `kogito-*`. > > > > > Luckily, we didn't have many incompatibilities, but right now we > have > > > > > an important one, that we'll discuss below. > > > > > > > > > > Note that at this point, and for this issue in particular, it is > > > > > unimportant to discuss why things were done the way they were, or > what > > > > > we should've done instead. I'm laying out the facts so that we can > > > > > understand how we're moving forward now. Of course, I would > personally > > > > > love to debate this on a separate thread :) > > > > > > > > > > --- > > > > > > > > > > One of the most impactful features of KIE Sandbox is the DMN > Runner. > > > > > It allows people to rapidly interact with their Decisions as they > > > > > author them, and this was all made possible because of the JIT > > > > > Executor (hosted at `kogito-apps`). KIE Sandbox also started to > allow > > > > > users to deploy their Decisions to OpenShift, then later to > > > > > Kubernetes, and now recently we allow users to deploy anything to > > > > > OpenShift and Kubernetes, with a very high level of customization. > All > > > > > these features depend on `kogito-*`, Quarkus, and all the things > that > > > > > come with them. Including the Java version. > > > > > > > > > > Logically, `kie-tools` depends on `drools` and `kogito-*`, as > `drools` > > > > > and `kogito-*` can (and do!) exist and function without Tools, but > not > > > > > the other way around. > > > > > > > > > > As `kie-tools` started growing, maturing, and releasing some > pieces of > > > > > technology that were useful to other domains (i.e. Multiplying > > > > > Architecture), it became a dependency for other projects. Users > > > > > started depending on our Standalone Editors, but also on the > > > > > Multiplying Architecture packages. One good example is Kaoto, which > > > > > released a VS Code extension aligned with our architecture. > > > > > > > > > > The capability of encapsulating web components in any technology > > > > > behind a well defined, statically typed API was really appealing, > and > > > > > when the Kogito initiative started to design their Runtime Tools > > > > > (i.e., Quarkus Dev UIs, Management Console, and Task Inbox), using > the > > > > > Multiplying Architecture was the way to go. > > > > > > > > > > That's when the circular dependency started. We had > > > > > `kogito-apps/ui-packages` depending on `kie-tools`. Note that it > was > > > > > also logical for these packages to be part of the `kogito-*` > > > > > repositories and release stream. Users upgrading to a new version > of > > > > > Kogito would also need to have their Task Inbox, Management > Console, > > > > > and Quarkus Dev UIs aligned with the new version of Kogito. Not to > > > > > mention Job Service, Data Index etc. > > > > > > > > > > For many months (or years!), this was not an issue, as the > Multiplying > > > > > Architecture packages were very stable and very rarely were > modified. > > > > > The last big change we had was 3 years ago, with the introduction > of > > > > > Shared Values [1], which did not impact the fire-and-forget and > > > > > Promise-based mechanisms we already had in place. > > > > > > > > > > Although we were aware of this circular dependency from the > beginning, > > > > > it was not that big of a deal for `kogito-apps` to depend on an > older, > > > > > dephased version of `kie-tools`. We didn't have plans of unifying > the > > > > > streams anyway, and it was the best solution for our users to have > > > > > their Runtime Tools packages released together with the Kogito > stream. > > > > > > > > > > That of course changed after the migration to Apache. > > > > > > > > > > --- > > > > > > > > > > If you reached this point, thank you for your interest in a little > bit > > > > > of the history of KIE, and I hope you can now understand why things > > > > > are the way they are. Let's talk about what changed and what we are > > > > > doing to adapt to the new requirements. > > > > > > > > > > Hinting a little bit to what I mentioned above, this discussion is > > > > > very intertwined with the CI pipelines and versioning/releasing > > > > > strategies subjects, as we are essentially discussing code > > > > > organization, coupling/decoupling and granularity of stuff, and to > > > > > make things easier at least for a while, `drools` and `kogito-*` > will > > > > > continue being a separate stream from `kie-tools`. > > > > > > > > > > `drools` and `kogito-*` will continue being able to evolve > > > > > independently of `kie-tools`, and `kie-tools` will continue > adapting > > > > > itself to whatever changes on `drools` and `kogito-*`. > > > > > > > > > > At the same time, Apache KIE 10 (and the next releases too!) will > be > > > > > atomic. And for our users, a single release stream. `drools@10` > and > > > > > `kogito-*@10` will be used on `kie-tools@10`. We started this > effort a > > > > > while ago, and although we should've been more open about it from > the > > > > > beginning, we are also learning how to properly communicate inside > the > > > > > Apache structure. Better late than never :) Thanks for the tip > Tibor! > > > > > > > > > > Here's how we'll do it: > > > > > > > > > > 1. A couple of weeks ago, one of our newest committers. Rodrigo > > > > > Antunes, created automated jobs to weekly publish a timestamped > > > > > SNAPSHOT version of `drools` and `kogito-*`. This will allow us to > > > > > have a persistent, immutable version to point to on `kie-tools`, > even > > > > > without having a full release. Timestamped SNAPSHOTs, as we're > calling > > > > > them, are a little bit different than normal Maven SNAPSHOTs, as > there > > > > > is no `updatePolicy`, so as time passes, the downloaded artifacts > do > > > > > not change. This allows `kie-tools` to be stable and incorporate > > > > > changes done to `drools` and `kogito-*` at its own pace. We also > moved > > > > > the publication of `@kie-tools/jitexecutor-native` to this new > job, so > > > > > we won't have to maintain it on the `kie-tools` side anymore. > Thanks > > > > > Rodrigo! > > > > > > > > > > 2. When we define the code freeze prior to releasing Apache KIE, > > > > > `drools` and `kogito-*` will need to be frozen a little bit earlier > > > > > than `kie-tools`, leaving a buffer for `kie-tools` to start > pointing > > > > > to the latest timestamped SNAPSHOT and adapting itself to the > changes. > > > > > My initial thoughts are around 2 weeks, but with constant updates > and > > > > > proper alignment between the two streams, I see this time reducing > to > > > > > a couple of days for upcoming releases. > > > > > > > > > > 3. After `kie-tools` is fully functional pointing to the latest > > > > > timestamped SNAPSHOT of `drools` and `kogito-*`, we can now start > > > > > preparing for the actual release. We will cut new branches for the > > > > > next version (i.e. 10.0.0) and update all configuration files that > we > > > > > have to. This will be done on `drools`, `kogito-*`, and > `kie-tools`. > > > > > Every module/package will declare the same version, and will > depend on > > > > > the upstream modules/packages with the same version as well. We can > > > > > proceed with this staging phase as normal. Running automated/manual > > > > > tests and eventually signing off that everything is good to go. > (I'm > > > > > not 100% on the timeline for the Apache release, but with this > process > > > > > we can guarantee that we will vote on branches that won't change > when > > > > > we decide to release. It will be a matter of pushing tags pointing > to > > > > > these release branches). > > > > > > > > > > With the release process defined, we can now discuss the exact > actions > > > > > we're taking to remove the circular dependency we have between > > > > > `kie-tools` and `kogito-apps`. All of which will need to be > addressed > > > > > prior to starting the release process. > > > > > > > > > > 1. (Fabrizio - IN PROGRESS) Make sure `kie-tools` is on Java 17, > Maven > > > > > 3.9.6, and Quarkus 3. > > > > > (https://github.com/apache/incubator-kie-issues/issues/960) > > > > > There's a complication here, since we still have three big > > > > > packages (Serverless Workflow Diagram Editor, BPMN, DMN, and SceSim > > > > > Editors, and DashBuilder) on `kie-tools` that need GWT, and as you > > > > > know, GWT together with Java upgrades usually require some work. > We're > > > > > on GWT 2.10, which should already support Java 17, but we'll know > more > > > > > once we try it. > > > > > > > > > > 2. (Fabrizio - NOT YET STARTED) Make sure `kie-tools` is using the > > > > > latest timestamped SNAPSHOT from `kogito-*`. > > > > > (https://github.com/apache/incubator-kie-issues/issues/965) > > > > > We have them built and published every Sunday EOD, so on Monday > > > > > (Feb 25th, 2024) we'll have a brand new one to start working with. > > > > > > > > > > 3. (Thiago - IN PROGRESS) Move `kogito-apps/ui-packages` to > > > > > `kie-tools`. ( > > > https://github.com/apache/incubator-kie-issues/issues/833) > > > > > We started a couple of weeks ago, and we should be ~80% done > with > > > it > > > > > by now. > > > > > > > > > > 4. (Fabrizio - PR SENT, but blocked by 1. and 2.) Move the SWF > Quarkus > > > > > Dev UI package from `kogito-apps` to `kie-tools`. > > > > > (https://github.com/apache/incubator-kie-tools/pull/2167) > > > > > This can only be done after 1. and 2. are merged, since this > move > > > > > will require `kie-tools` to be ready to build packages with Java > 17, > > > > > Maven 3.9.6, and Quarkus 3. > > > > > > > > > > 5. (Thiago - NOT YET STARTED) Move the BPMN Quarkus Dev UI package > > > > > from `kogito-apps` to `kie-tools`. > > > > > (https://github.com/apache/incubator-kie-issues/issues/966) > > > > > We'll follow the lead of Fabrizio's PR from 4. That's > essentially > > > > > doing exactly the same thing for the BPMN Quarkus Dev UI module. > > > > > > > > > > 6. (Rodrigo - NOT YET STARTED) Change the Jenkins release jobs to > > > > > include publishing both Quarkus Dev UI modules to Maven central as > > > > > part of the Apache release. > > > > > (https://github.com/apache/incubator-kie-issues/issues/964) > > > > > Should be pretty straight-forward, as we already have > experience > > > > > doing this for other repos. > > > > > > > > > > 7. (Thiago - NOT YET STARTED) Change the ownership of > `task-console` > > > > > and `management-console` images from `kogito-images` to > `kie-tools`. > > > > > (https://github.com/apache/incubator-kie-issues/issues/963) > > > > > KIE Tools already hosts and publishes several images, and since > > > > > the content of those images will now be hosted at `kie-tools`, we > can > > > > > only host them there. > > > > > This also means we'll get rid of the two Quarkus apps [2] that > > > > > wrap both consoles' static assets. We could be wrong about the > purpose > > > > > of those apps, so if anyone has more information, please do reach > out. > > > > > Otherwise, we'll skip the Quarkusification of those static assets > and > > > > > we'll ship images with `httpd` directly, like we do for KIE > Sandbox. > > > > > > > > > > 8. (Assignee TDB - NOT YET STARTED) Deleting everything that was > moved > > > > > to `kie-tools` from `kogito-apps`. > > > > > (https://github.com/apache/incubator-kie-issues/issues/962) > > > > > That will be the point where we actually get rid of the > circular > > > > > dependency and remove the blocker for the release. > > > > > > > > > > 9. (Assignee TDB - NOT YET STARTED) Adapt `kogito-examples` to all > the > > > > > changes we mentioned above. > > > > > (https://github.com/apache/incubator-kie-issues/issues/961) > > > > > Kogito Examples keeps being part of the `drools` and `kogito-*` > > > > > stream, meaning that example modules won't be able to use neither > of > > > > > the Quarkus Dev UIs, as `kie-tools` is downstream in relation to > the > > > > > `drools` and `kogito-*` stream. > > > > > The same would be true for Management Console and Task Console > > > > > images, but for them, we can use the `daily-dev` tag we publish > daily > > > > > from `kie-tools@main`. > > > > > > > > > > 10. (Assignee TDB - NOT YET STARTED) Create some example apps that > use > > > > > the Quarkus Dev UIs directly on the `examples` directory of > > > > > `kie-tools`. ( > > > https://github.com/apache/incubator-kie-issues/issues/959) > > > > > This is to serve the same purpose of packages that did that on > > > > > `kogito-examples`, but now hosted on `kie-tools`. > > > > > > > > > > Pheew! That's a lot of things to do, but it is the result of two > > > > > intense weeks assessing and planning how we would address this > issue > > > > > in a way that the Apache KIE 10 release can be done in a way that > > > > > feels atomic to our users, who are ultimately the most important > part > > > > > of the community. There's an open EPIC too at > > > > > https://github.com/apache/incubator-kie-issues/issues/967. > > > > > > > > > > I appreciate it if you reached this point, and hope you feel > welcome > > > > > to contributing to solving this lengthy and historical problem if > you > > > > > feel like doing so. > > > > > > > > > > I'll personally send updates to this thread every week, on Thursday > > > > > nights. Do not hesitate to contact me or any of the people > involved in > > > > > this effort if you have questions, or simply want to chat about it. > > > > > > > > > > Thanks a lot Alex, Paulo, Fabrizio, Thiago, Rodrigo, and others > that > > > > > somehow helped so far. > > > > > > > > > > Kind regards, > > > > > > > > > > Tiago Bento > > > > > > > > > > --- > > > > > > > > > > P.S.: You might've noticed we left Trusty out of the picture for > now. > > > > > Thiago is conducting an investigation to understand what would be > the > > > > > best course of action for it. As much as we have plans to revive > it, > > > > > right now our focus has to be on getting Apache KIE 10 out. We can > > > > > always rely on Git to bring it back in the future, if we ultimately > > > > > decide that wiping out Trusty is the best strategy for this moment. > > > > > > > > > > [1] https://github.com/apache/incubator-kie-tools/pull/691 > > > > > > > > > > [2] > > > > > > > > > https://github.com/apache/incubator-kie-kogito-apps/tree/main/task-console > > > > > and > > > > > > > > > https://github.com/apache/incubator-kie-kogito-apps/tree/main/management-console > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
