On Thu, Jun 1, 2017 at 8:17 AM, Timo Goebel <[email protected]> wrote:
> > Am Donnerstag, 1. Juni 2017 12:17:49 UTC+2 schrieb Lukas Zapletal: >> >> I would like to open discussion about having a unit test job of >> foreman with all major plugins enabled (katello, discovery, bootdisk, >> rex, openscap). >> > > What would be all the major plugins? Expire Hosts? Digitalocean? Omaha? > > The list gets quite long easily. How about making it a responsibility of a > plugin to make sure it actually works? If as a maintainer of the omaha > plugins I want a nightly job to test the plugin with more plugins enabled, > nobody stops me from setting one up. > I'd also disable the katello ci job in core to be honest. It should not be > the responsibility of core to make sure specific plugins work, but the > responsibility of the plugins. > This sounds good in theory, but I'll refresh the history of why we did this. A change would be made to Foreman core, that change would go in and then propagate out. A developer would update their foreman checkout or the Katello pipeline would run and break. More developers start updating local checkouts and now everyone is broken. The Katello PR tests begin to fail and all PRs are frozen. So now a significant portion of developers are all trying to figure out what point in time to roll back to, who is going to fix the breakage and how long it is going to take. 1-5 days later, the issue is fixed and PRs are able to start back up as well as the nightly pipeline. How much developer frustration was created? How much time was lost and wasted? If we want core to have more freedom of movement then core needs to evolve to allow plugins more reliability on interfacing. Not all plugins are equal in size, developer community or usage and thus they should not be all treated the same in my opinion. It is my opinion that we need to accept this formally and as others have said build our testing and pipelines around this or change our model for adding functionality. Sometimes it is like we are adverse to accepting we have evolved to have a common stack of "core" plugins that would benefit from being treated as such to enhance reliability, testing, hardening. Today, the only gating we have is that blessing that is Katello unit tests being run on Foreman PRs. The nightly pipeline has no notion of the plugins and routinely sends out changes that break Katello and other plugins making the plugin repo and Katello nightly repository a crap shoot with respect to will it install. I am all for more testing, but what good does testing do if there are no consequences? A nightly test that runs and either passes or fails is informative and requires someone to be looking and willing to act on every change. If the test has no consequence on deployment that likelihood drops dramatically in my opinion. I think about this a lot and how it would be ideal if we came up with a gating and release system that worked for each plugin. Pushing new versions, or nightlies if and only if they pass some testing that way we knew that everything in nightly worked. Or accepting we are a stack, having a system whereby plugins could be accepted as a tier 1 type status that included them in stack testing and blocked with some agreement and understanding from the developers that they have to address issues in a timely manner. Maybe we need to get rid of this plugin model all together and switch to a services model that allows more independent control and flexibility from a code writing, testing and deployment stand point. Or monolith all the things so we are a single community working on a common core. Another thing we have to consider is that we have people who work on and take care of the infrastructure but we do not have an infrastructure or CI/CD team or SIG that takes responsibility for and thinks about these complex problems. I don't know how we reliably add more without addressing this. Half my two cents, Eric > I think, it would be a better approach to have a nightly job that tests > the "Satellite 6" configuration to check if everything still plays nice > together. > > - Timo > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- Eric D. Helms Red Hat Engineering -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
