Tiago already provided a in-depth explanation of the issue in this ML, that is summarized in the circular dependency thread [1]
TL;DR: **Today** KIE-Tools is not part of any PR checks in the runtime repositories (drools, optaplanner, runtimes and apps) - so a change in those repos can break KIE Tools. If we were to continue to use SNAPSHOTs, we need unify to a single CI that would include all repositories - including KIE Tools. [1] - https://lists.apache.org/thread/58xm7pqdyztf7qztmhvntf8wdmvfx7jx On Fri, May 10, 2024 at 5:15 AM Francisco Javier Tirado Sarti <ftira...@redhat.com> wrote: > > Hi Tiago, > I would argue that, being all the repos part of the same release, mutable > snapshot is perfectly acceptable. As it currently happens between Runtimes > repo and Drools repo (which have the Kie api) and Apps repo and Runtimes > repo. I, as I believe Enrique (at least if what I understood from his > e-mail) , do not understand why tooling should be different in that regard. > > On Fri, May 10, 2024 at 1:26 AM Tiago Bento <tiagobe...@apache.org> wrote: > > > Zanini and Enrique, > > > > Not sure if I'm making it sound differently, but everything I'm saying > > is also aiming towards a healthy CI. Not only for SonataFlow or Tools, > > though. For all of Apache KIE. > > > > My main point is that mutable Maven SNAPSHOTs are harming us, giving a > > false impression of synchronization between the repos. It's an "easy > > way out" we've been historically abusing, IMHO. > > > > One of the best advantages of having multiple repos is that they can > > be individually developed, tested, and managed. However, by using > > Maven SNAPSHOTs as the synchronization mechanism, we can't guarantee > > that, since repos become broken on any given day. More than that, it's > > impossible to go "back in time" on our development branches, since the > > SNAPSHOT version will always point to latest. > > > > Please read this section from the guidelines of the SciJava project at > > https://github.com/scijava/pom-scijava-base/blob/main/README.md. > > > > "Reproducible builds. This rule means no SNAPSHOT dependencies, no > > SNAPSHOT parents, and no SNAPSHOT plugin versions. A snapshot version > > is not immutable, which means that code which depends on a snapshot > > may build today, but not build tomorrow, if the snapshot is later > > changed. The best way to avoid this conundrum is to never depend on > > SNAPSHOT versions. Snapshot are best used for testing only; they can > > be used transiently, but their use should never make it onto the main > > integration branch (e.g., main or master) of a project. See also Using > > snapshot couplings during development at > > > > https://imagej.net/develop/architecture#using-snapshot-couplings-during-development > > ." > > > > This summarizes very well the problem I'm trying to convey here. Of > > course, their words are much more clear than all the explanations I > > tried giving so far. > > > > > > > > > > On Thu, May 9, 2024 at 9:43 AM ricardo zanini fernandes > > <ricardozan...@gmail.com> wrote: > > > > > > Tiago, > > > > > > Again, this is a major problem in our current structure that we should > > use > > > another thread to discuss. For my proposal is to at least have a health > > > cloud build platform. We can completely remove UI from our images in the > > > SonataFlow space and keep the consoles in kie-tools. Once we solve the > > deep > > > structural problem we have, we can evaluate it better. What we can't do > > is > > > be behind and not release. > > > > > > The template image is pretty useful if you really want to test the DevUI > > in > > > the image. You can easily pull the container file and build the image > > > during your integration tests and run the tests you need. We can do the > > > same in the image. I don't see a waste of resources, but as a temporary > > > alternative until we figure out the deeper problem. > > > > > > At the moment, everything "tools" related is in a single repo and the > > other > > > components are scattered in many other repos. We have to decide if we > > want > > > mono repos (infra-oriented) or context repos (feature-oriented). We can't > > > have both otherwise the mono repos will start to drag others to it, as we > > > are seeing happening. > > > > > > So IMO, we need first to sit down together and have a new repo structure > > > design, perhaps based on components. We need to evaluate. > > > > > > For the time being, we need to move forward at least and I can live with > > > integration tests at the end of the line using final versions/built > > > versions of the DevUI if needed for SonataFlow. > > > > > > > > > > > > On Thu, May 9, 2024 at 10:32 AM ricardo zanini fernandes < > > > ricardozan...@gmail.com> wrote: > > > > > > > Hi Enrique! > > > > > > > > The images are using SNAPSHOTS from apps/runtimes currently. There was > > a > > > > decision in the past to build apps in the images from the > > main/versioned > > > > branch. We are changing this to use SNAPSHOT/versioned artifacts > > published > > > > in Maven. About kie-tools, I don't know either. > > > > > > > > We can use immutable snapshots (weekly, nightlies) instead. I'm not > > > > against it. But we do need those for integration tests or at least a > > robust > > > > CI platform to do integration without snapshots. > > > > > > > > Cheers! > > > > > > > > On Thu, May 9, 2024 at 5:07 AM Enrique Gonzalez Martinez < > > > > egonza...@apache.org> wrote: > > > > > > > >> Hi Tiago, > > > >> > > > >> I think the discussion is related to the graph dependency for now. So > > > >> I would keep that in mind. Anyway I am kinda surprised about some > > > >> concepts like (mutable snapshots and using snapshot published > > > >> timestamps like they are final). > > > >> > > > >> Now I think I understand where the problem lies now; so basically if > > > >> we had the proper graph dependency project built based on proper > > > >> snapshots everything would build just ok. So my question is pretty > > > >> simple, why is it not possible for kie tools or kogito images to use > > > >> the maven snapshot concept and point out to the last snapshot build > > > >> (either local or remote) like other maven projects work. Is there any > > > >> limitation there ? > > > >> > > > >> El mié, 8 may 2024 a las 22:30, Tiago Bento (<tiagobe...@apache.org>) > > > >> escribió: > > > >> > > > > >> > Hi Zanini! Welcome back, hope you had a great time during your PTO. > > > >> > > > > >> > If I understand correctly, the way you see the integration between > > > >> > repos relies heavily on timestamped SNAPSHOTs. This way: > > > >> > 1. `kogito-images` would stop building `kogito-apps`, and would rely > > > >> > on timestamped SNAPSHOTs being published from > > > >> > `drools/optaplanner/kogito-runtimes/kogito-apps`. > > > >> > 2. `kie-tools` would start publishing timestamped SNAPSHOTs for its > > UI > > > >> > components, so they would be consumed downstream by > > > >> > `kogito-serverless-operator`. > > > >> > > > > >> > If that's what you mean, let me elaborate on some of the > > consequences > > > >> > of this integration model, where we stop using the mutable > > > >> > 999-SNAPSHOT and start relying more on timestamped SNAPSHOTs for > > > >> > cross-repo synchronization: > > > >> > a. Cross-repo PRs would not be possible anymore, depending on which > > > >> > repos are involved in the ensemble. This would create boundaries > > > >> > between certain groups of our repos, namely: > > > >> > drools/optaplanner/kogito-runtimes/kogito-apps --> kogito-images --> > > > >> > kie-tools, --> kogito-serverless-operator. So 4 "independent > > > >> > clusters". > > > >> > b. kogito-images, kie-tools, and kogito-serverless-operator, using > > > >> > timestamped SNAPSHOTs between themselves, would make the most > > upstream > > > >> > of the three (in this case, `kogito-images`), to be the one defining > > > >> > what timestamped SNAPSHOT version we should use for its upstream > > > >> > dependencies (drools/optaplanner/kogito-runtimes/kogito-apps). In > > more > > > >> > details: If `kogito-images` depends on > > > >> > `drools/optaplanner/kogito-runtimes/kogito-apps` using version > > > >> > 999-20240501, that's the version `kie-tools` and > > > >> > `kogito-serverless-operator` need to use as well, as the > > > >> > drools/optaplanner/kogito-runtimes/kogito-apps would come > > transitively > > > >> > from the `kogito-images` dependency these two repos would have. > > This, > > > >> > IMHO, causes a lot of confusion, since the dates on the timestamped > > > >> > SNAPSHOT versions have implicit transitive dependencies to other > > > >> > upstream projects that also have their own timestamped SNAPSHOTs > > > >> > published. I.e., 999-20240501 of `kie-tools` won't be aligned with > > > >> > 999-2024051 of `drools/optaplanner/kogito-runtimes/kogito-apps`. > > > >> > c. Updating dependencies across the board would be much more > > > >> > difficult, as we would need to rely on a chain reaction starting > > from > > > >> > the `drools/optaplanner/kogito-runtimes/kogito-apps` cluster all the > > > >> > way down to `kogito-serverless-operator`, releasing timestamped > > > >> > SNAPSHOTs and sending PRs for each "cluster" along the way. > > > >> > > > > >> > I'm not sure this is the way we want to move forward, to be honest, > > as > > > >> > these consequences need to be thoroughly examined by people in the > > > >> > community, so that all the side-effects of this choice are clear to > > > >> > everyone. > > > >> > > > > >> > To conclude, having `kie-tools` building a "template image" to > > verify > > > >> > that it will not break `kogito-serverless-operator` downstream, and > > > >> > having `kogito-serverless-operator` building parts of `kie-tools` to > > > >> > incorporate the latest changes, IMHO, is: > > > >> > 1. Duplication of building logic. IMHO, we can't have individual > > repos > > > >> > creating its own up- and downstream CI systems. > > > >> > 2. Conflicting with the timestamped SNAPSHOTs strategy for > > integrating > > > >> > repos, where boundaries are clearly defined. > > > >> > 3. Wasteful use of resources (same that is happening with > > > >> > `kogito-images` building `kogito-apps` right now). > > > >> > > > > >> > I'm sorry for bringing all this to this seemingly simple proposal, > > but > > > >> > I'm afraid taking steps without knowing where we're going isn't the > > > >> > way to move forward. > > > >> > > > > >> > Please let me know if I misunderstood something. > > > >> > > > > >> > On Wed, May 8, 2024 at 3:15 PM ricardo zanini fernandes > > > >> > <ricardozan...@gmail.com> wrote: > > > >> > > > > > >> > > Hi! > > > >> > > > > > >> > > Sorry for the late reply, I was on PTO. > > > >> > > > > > >> > > Replies inline. > > > >> > > > > > >> > > > The `kie-tools` CI is not prepared to build > > > >> > > `kogito-serverless-operator`, so the way > > `kogito-serverless-operator` > > > >> > > references the SonataFlow Quarkus Dev UI will be important to > > > >> > > establish the boundaries between both repos. To further develop > > the > > > >> > > SonataFlow Quarkus Dev UI and have its changes reflect on the > > > >> > > `kogito-swf-devmode` image, we need to come up with a strategy > > that is > > > >> > > both safe, consistent, and enforces correctness. There's also the > > fact > > > >> > > that currently `kie-tools` depends on timestamped SNAPSHOTs of > > > >> > > Kogito/Drools, while `kogito-serverless-operator` uses > > 999-SNAPSHOT, > > > >> > > if I'm not mistaken. Can you elaborate a little bit more on how > > you > > > >> > > see this reference being done? Please consider cross-repo PRs for > > big > > > >> > > migrations like the Quarkus 3.8.4 that is currently happening. > > > >> > > > > > >> > > This will make us discuss yet again the need of snapshots for the > > UI > > > >> > > components used by the images. The images must be the end of the > > build > > > >> > > pipeline, where we aggregate every component we ship in a single > > > >> instance > > > >> > > to release. Yet, they are a core part of the cloud platform, hence > > > >> part of > > > >> > > the Operator BDD, testing, integration, and delivering. > > Intermediate > > > >> repos > > > >> > > can't depend on them unless they are also responsible for > > maintenance > > > >> and > > > >> > > release like what we're doing with the consoles. > > > >> > > > > > >> > > Also, the images can have a respin anytime we need a new component > > > >> bump. We > > > >> > > do this all the time to fix CVEs or bug fixes in a specific > > component > > > >> that > > > >> > > is part of the image. > > > >> > > > > > >> > > The devui can be developed and maintained within the kie-tools > > repo > > > >> and > > > >> > > have tests that verify that component. The integration will be > > made > > > >> on the > > > >> > > Images/Operator side once we grab a latest snapshot. > > Alternatively, > > > >> we can > > > >> > > do: > > > >> > > 1. The tools CI can fetch a template image, inject the component, > > and > > > >> run > > > >> > > the required tests > > > >> > > 2. The image/operator CI can fetch and build the UI components and > > > >> run the > > > >> > > integration tests > > > >> > > > > > >> > > > Also, regarding point "4. Review the PR GHA checks to run CLI > > tests > > > >> > > once there's a change in the cmd module" of the proposed EPIC, I > > think > > > >> > > we might run into problems, since the `cmd` module also depends > > on the > > > >> > > `api` and `workflowproj` modules of `kogito-serverless-operator.` > > I'm > > > >> > > afraid changes made to these two modules would also need to > > trigger a > > > >> > > build of the `cmd` module, and they can potentially break it. > > > >> > > > > > >> > > Sorry to not get into details there, but the new cmd module will > > be > > > >> part of > > > >> > > the Operator's workspace, any changes in any cross dependencies > > there > > > >> will > > > >> > > trigger the CI checks, as we are doing today in the builder and > > > >> > > workflowproj modules. It will be way easier to maintain the CLI. > > > >> > > > > > >> > > > These considerations alone, IMHO, expose one of the biggest > > > >> challenges > > > >> > > we have in our community right now, which is that the definition > > and > > > >> > > implementation of the dependency graph between > > repos/modules/packages > > > >> > > is currently spread across many different "build systems", like > > the > > > >> > > new proposed GHA jobs exclusive to the > > `kogito-serverless-operator` > > > >> > > repo, the Build Chain system we have for the Drools/Kogito repos, > > the > > > >> > > `kie-tools` CI, and the many Jenkins jobs we have on Apache's > > Jenkins. > > > >> > > There's also the fact that we have `kogito-images` selectively > > > >> > > building parts of `kogito-apps` during its own build to include > > them > > > >> > > in the images it produces. > > > >> > > > > > >> > > I agree. We would need to sit together and solve this as a team, > > > >> having a > > > >> > > nice integration across every repo. The kogito-images building > > parts > > > >> of the > > > >> > > apps is something we currently working on, IIRC. The apps will > > deploy > > > >> a > > > >> > > snapshot as part of the nightlies and the images will be using it. > > > >> > > > > > >> > > > I just think it is important to highlight that this proposal > > would > > > >> > > only address a LOCAL problem related exclusively to the SonataFlow > > > >> > > section of the Apache KIE community, while, at the same time, not > > > >> > > contributing to reducing the segmentation of the Apache KIE > > community > > > >> > > as a whole. > > > >> > > > > > >> > > Yes, the proposal is to starting addressing the local Apache KIE > > > >> SonataFlow > > > >> > > Cloud platform first. But I agree that we need to refactor our CI > > as a > > > >> > > whole, which is something we should start in a new thread. > > > >> > > > > > >> > > Cheers! > > > >> > > > > > >> > > On Mon, Apr 22, 2024 at 10:02 PM Tiago Bento < > > tiagobe...@apache.org> > > > >> wrote: > > > >> > > > > > >> > > > Zanini and Alex, > > > >> > > > > > > >> > > > The task we agreed on for after releasing Apache KIE 10 is > > > >> > > > https://github.com/apache/incubator-kie-issues/issues/1040. It > > only > > > >> > > > describes deleting the temporary copies we'll have on KIE Tools > > and > > > >> > > > reverting things back to where they were, using the fixed 10.0.0 > > > >> > > > version. > > > >> > > > > > > >> > > > Moving `kn-plugin-workflow` and the > > `kogito-swf-{devmode,builder}` > > > >> > > > images to `kogito-serverless-operator` would be a new move, > > which I > > > >> > > > understand is the scope of this proposal thread. > > > >> > > > > > > >> > > > Doing so, however, will make `kogito-serverless-operator` > > depend on > > > >> > > > `kie-tools`, since the SonataFlow Quarkus Dev UI is hosted there > > > >> now, > > > >> > > > and it is a dependency of the `kogito-swf-devmode` image. > > > >> > > > > > > >> > > > I'm saying this because I think we need to further discuss the > > > >> > > > consequences of this change... > > > >> > > > > > > >> > > > The `kie-tools` CI is not prepared to build > > > >> > > > `kogito-serverless-operator`, so the way > > > >> `kogito-serverless-operator` > > > >> > > > references the SonataFlow Quarkus Dev UI will be important to > > > >> > > > establish the boundaries between both repos. To further develop > > the > > > >> > > > SonataFlow Quarkus Dev UI and have its changes reflect on the > > > >> > > > `kogito-swf-devmode` image, we need to come up with a strategy > > that > > > >> is > > > >> > > > both safe, consistent, and enforces correctness. There's also > > the > > > >> fact > > > >> > > > that currently `kie-tools` depends on timestamped SNAPSHOTs of > > > >> > > > Kogito/Drools, while `kogito-serverless-operator` uses > > 999-SNAPSHOT, > > > >> > > > if I'm not mistaken. Can you elaborate a little bit more on how > > you > > > >> > > > see this reference being done? Please consider cross-repo PRs > > for > > > >> big > > > >> > > > migrations like the Quarkus 3.8.4 that is currently happening. > > > >> > > > > > > >> > > > Also, regarding point "4. Review the PR GHA checks to run CLI > > tests > > > >> > > > once there's a change in the cmd module" of the proposed EPIC, I > > > >> think > > > >> > > > we might run into problems, since the `cmd` module also depends > > on > > > >> the > > > >> > > > `api` and `workflowproj` modules of > > `kogito-serverless-operator.` > > > >> I'm > > > >> > > > afraid changes made to these two modules would also need to > > trigger > > > >> a > > > >> > > > build of the `cmd` module, and they can potentially break it. > > > >> > > > > > > >> > > > These considerations alone, IMHO, expose one of the biggest > > > >> challenges > > > >> > > > we have in our community right now, which is that the > > definition and > > > >> > > > implementation of the dependency graph between > > > >> repos/modules/packages > > > >> > > > is currently spread across many different "build systems", like > > the > > > >> > > > new proposed GHA jobs exclusive to the > > `kogito-serverless-operator` > > > >> > > > repo, the Build Chain system we have for the Drools/Kogito > > repos, > > > >> the > > > >> > > > `kie-tools` CI, and the many Jenkins jobs we have on Apache's > > > >> Jenkins. > > > >> > > > There's also the fact that we have `kogito-images` selectively > > > >> > > > building parts of `kogito-apps` during its own build to include > > them > > > >> > > > in the images it produces. > > > >> > > > > > > >> > > > With all that said, I'm not opposed to moving the > > > >> `kn-workflow-plugin` > > > >> > > > package from `kie-tools` to the `cmd` module of > > > >> > > > `kogito-serverless-operator`. In fact, like I said in the past, > > I > > > >> > > > think it makes a lot of sense that they're part of the same > > > >> > > > compilation unit, as they're very closely related, and should > > > >> > > > therefore be in sync at all times. > > > >> > > > > > > >> > > > I just think it is important to highlight that this proposal > > would > > > >> > > > only address a LOCAL problem related exclusively to the > > SonataFlow > > > >> > > > section of the Apache KIE community, while, at the same time, > > not > > > >> > > > contributing to reducing the segmentation of the Apache KIE > > > >> community > > > >> > > > as a whole. > > > >> > > > > > > >> > > > Regards, > > > >> > > > > > > >> > > > Tiago Bento > > > >> > > > > > > >> > > > On Fri, Apr 19, 2024 at 4:08 PM ricardo zanini fernandes > > > >> > > > <ricardozan...@gmail.com> wrote: > > > >> > > > > > > > >> > > > > Alex, > > > >> > > > > > > > >> > > > > Yes, in the proposal we just barely outlined. I create the > > EPIC > > > >> to have > > > >> > > > > more details and start working on them as soon as we agree. > > > >> > > > > > > > >> > > > > On Fri, Apr 19, 2024 at 4:24 PM Alex Porcelli < > > a...@porcelli.me> > > > >> wrote: > > > >> > > > > > > > >> > > > > > Thank you for outlining the tasks post the 10.x release. > > It's > > > >> > > > > > important to note that these are already included in the > > amended > > > >> > > > > > proposal [1], specifically in steps 9 and 10, which the > > > >> community has > > > >> > > > > > voted on. If there are concerns about the execution of these > > > >> steps, > > > >> > > > > > I'd like to reassure you that they will proceed as planned, > > in > > > >> > > > > > compliance with Apache guidelines. > > > >> > > > > > > > > >> > > > > > Looking ahead, after the release of version 10, we already > > > >> agreed that > > > >> > > > > > we'll need to have a thorough discussion regarding the > > codebase > > > >> > > > > > structure. This will allow us to refine our understanding > > of the > > > >> > > > > > sub-brands, their interrelationships, and their strategic > > > >> positioning. > > > >> > > > > > I agree that this is crucial for our next steps and look > > > >> forward to > > > >> > > > > > our collaborative efforts in shaping this. > > > >> > > > > > > > > >> > > > > > [1] - > > > >> https://lists.apache.org/thread/pw327lkpmy9gxklq4t5zbwzxjo2y3c0w > > > >> > > > > > > > > >> > > > > > On Fri, Apr 19, 2024 at 2:50 PM ricardo zanini fernandes > > > >> > > > > > <ricardozan...@gmail.com> wrote: > > > >> > > > > > > > > > >> > > > > > > Folks, > > > >> > > > > > > > > > >> > > > > > > I've outlined the tasks we need once we release 10.x from > > > >> kie-tools: > > > >> > > > > > > > > https://github.com/apache/incubator-kie-issues/issues/1102 > > > >> > > > > > > > > > >> > > > > > > Once we release, we can detail this planning and start > > > >> working on it > > > >> > > > to > > > >> > > > > > > have a streamlined process for the next release. > > > >> > > > > > > > > > >> > > > > > > Please let me know if it makes sense to you. We can break > > > >> down and > > > >> > > > detail > > > >> > > > > > > the tasks once we agree on this initial plan. > > > >> > > > > > > > > > >> > > > > > > Cheers! > > > >> > > > > > > -- > > > >> > > > > > > Ricardo Zanini Fernandes > > > >> > > > > > > Vida longa e próspera. > > > >> > > > > > > > > >> > > > > > > > > >> --------------------------------------------------------------------- > > > >> > > > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > > >> > > > > > For additional commands, e-mail: dev-h...@kie.apache.org > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > >> --------------------------------------------------------------------- > > > >> > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > > >> > > > For additional commands, e-mail: dev-h...@kie.apache.org > > > >> > > > > > > >> > > > > > > >> > > > > >> > > > --------------------------------------------------------------------- > > > >> > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > > >> > For additional commands, e-mail: dev-h...@kie.apache.org > > > >> > > > > >> > > > >> --------------------------------------------------------------------- > > > >> To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > > >> For additional commands, e-mail: dev-h...@kie.apache.org > > > >> > > > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > For additional commands, e-mail: dev-h...@kie.apache.org > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org For additional commands, e-mail: dev-h...@kie.apache.org
