Re: [foreman-dev] Re: Unit test jenkins job with all plugins enabled

[Tomas Strachota]

On Mon, Jun 5, 2017 at 4:14 PM, Lukas Zapletal  wrote:
> On Mon, Jun 5, 2017 at 2:48 PM, Tomer Brisker  wrote:
>> My suggestion is that we replace the katello test currently running on every
>> PR with a test that installs the top 5 plugins together and runs all of
>> their tests.
>
> +1 +1 +1
>
> Last week or two, Katello was red because of trivial regression in the
> testing framework, we did not block PRs at all, but eventually Daniel
> stepped up and fixed it. That sounds to me this works! If we can
> extend this to other plugins - big improvement for these. For
> discovery this would be huge improvement in overall stability.
>
> I agree that it should not make the jenkins load much worse, discovery
> and also other plugins has only dozens of tests, so the slowdown
> should not be significant I hope. We can stick with what Katello team
> does today - only test postgresql on particular ruby version (2.2 I
> think today). If we are able to run them all in one rake task, that
> would mean just one Rails boot as well.

+1 for testing a set of most used plugins on postgresql and one ruby
version for each PR. Together with that we can still keep testing for
the full matrix once a week.

>
> What others say? Next steps would be:
>
> - nominate the plugins for this and vote them (or we can simple take
> most downloaded from our survey)
> - modify katello job to test all of them
> - rename katello job perhaps to avoid confusion?
>
> --
> Later,
>   Lukas @lzap Zapletal
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Re: Unit test jenkins job with all plugins enabled

[Tomer Brisker]

Hi,

It would be best, in my opinion, that we are able to tell as soon as
possible that some change will break plugins, and for that we'd need to run
tests on PRs.
My suggestion is that we replace the katello test currently running on
every PR with a test that installs the top 5 plugins together and runs all
of their tests.
We won't block merging PRs in core if that test fails, but at least we know
that the PR breaks plugins and be able to take action on it - either by
notifying maintainers, fixing the plugin directly, or making a change to
the PR so that they don't break.
This also has the added benefit of giving us a better indication that
plugins don't conflict with each other, which we currently don't test at
all afaik. Testing each plugin separately should be done by the plugins
themselves, not by core.
Most plugins' test are fairly short, so this won't add a significant load
to Jenkins over the current amount.

Tomer

On Mon, Jun 5, 2017 at 3:24 PM, Lukas Zapletal  wrote:

> We currently run plugin develop tests only on Sunday, I assume this
> has something to do with capacity planning. Today I stumbled upon
> another taxonomy related regression in develop which broke Discovery
> tests, I had to git bisect it until I found the commit in core,
> Jenkins was only able to tell me it was working on Sunday one week
> before. I'd love to see develop plugins tests being executed every day
> at least.
>
> On Fri, Jun 2, 2017 at 10:09 AM, Marek Hulán  wrote:
> > While I agree, one thing that properly defined plugin DSL does not solve
> is
> > the fact, that plugins now depend on each other. Few examples
> >
> > openscap, katello depends on remote execution
> > katello, remote execution, ansible, chef, ... depends on foreman tasks
> > remote execution depends on foreman templates (to some degree)
> > virt-who-configure depends on katello
> >
> > Recently, I was lucky to spot that one Katello PR was changing the
> Hypervisors
> > task API while virt-who plugin was subscribed to it, which would cause a
> > breakage. We managed to solve that before the PR was merged. Another
> example
> > is that foreman_templates changed import method interface, which caused
> remote
> > execution to fail importing templates so it needed to be fixed. The fix
> made
> > openscap and katello broken because they seed job templates. Openscap was
> > fixed in time but Katello 3.4 is now uninstallable if people have remote
> > execution enabled and we'll need 3.4.1 to fix it.
> >
> > I think we need to treat some plugins as core. If we can't merge them to
> core,
> > we should make sure, that their changes don't break other plugins.
> Therefore I
> > think we need some job that verifies that as many plugins as possible are
> > continuously green. In my opinion, people working on core usually can
> avoid
> > breaking changes if they know about it. If not, they should provide
> fixes or
> > at least guidance for affected plugins.
> >
> > --
> > Marek
> >
> > On čtvrtek 1. června 2017 18:47:05 CEST Eric D Helms wrote:
> >> On Thu, Jun 1, 2017 at 10:18 AM, Tomer Brisker 
> wrote:
> >> > Hello,
> >> >
> >> > While it is plugin maintainers responsibility to keep their plugins
> up to
> >> > date, one of the most common frustrations they face is "core changed
> and
> >> > broke something AGAIN". I wouldn't want to slow down core's already
> slow
> >> > merge process, but at the same time core contributors should try to
> >> > minimize breaking plugins as much as possible - for example, by
> >> > deprecating
> >> > functions before removing them completely. We could make these tests
> >> > optional - i.e. not block merge if plugin tests fail, but just take
> >> > advantage of the tests to fix whatever is needed in the plugins sooner
> >> > rather then later. I know that in certain cases having katello test
> run on
> >> > every PR let us find and fix issues we wouldn't have known about
> >> > otherwise.
> >> > Right now contributors (and reviewers) don't have any visibility to
> >> > whether a PR will cause problems to other plugins. If breaking is
> >> > unavoidable, at least having plugin tests run on PRs will allow
> giving a
> >> > "heads up" to plugin maintainers that they need to fix something,
> rather
> >> > then realizing after the next release that something broke.
> >> > During the past couple of months we had several cases of core PRs
> leading
> >> > to broken plugins, which could have been prevented or at least handled
> >> > better had we known that a certain change requires changes in plugins.
> >>
> >> Agree! If I had to sum it up, developer happiness and innovation at
> speed
> >> (core and plugins) is key! One option we've mentioned before that goes
> >> directly to Tomer's point is a stable plugin API. We as a community both
> >> develop and tell other developers, come, build a plugin. We have a
> vibrant
> >> plugin community. We say plugins are central to the 

Re: [foreman-dev] Re: Unit test jenkins job with all plugins enabled

[Lukas Zapletal]

We currently run plugin develop tests only on Sunday, I assume this
has something to do with capacity planning. Today I stumbled upon
another taxonomy related regression in develop which broke Discovery
tests, I had to git bisect it until I found the commit in core,
Jenkins was only able to tell me it was working on Sunday one week
before. I'd love to see develop plugins tests being executed every day
at least.

On Fri, Jun 2, 2017 at 10:09 AM, Marek Hulán  wrote:
> While I agree, one thing that properly defined plugin DSL does not solve is
> the fact, that plugins now depend on each other. Few examples
>
> openscap, katello depends on remote execution
> katello, remote execution, ansible, chef, ... depends on foreman tasks
> remote execution depends on foreman templates (to some degree)
> virt-who-configure depends on katello
>
> Recently, I was lucky to spot that one Katello PR was changing the Hypervisors
> task API while virt-who plugin was subscribed to it, which would cause a
> breakage. We managed to solve that before the PR was merged. Another example
> is that foreman_templates changed import method interface, which caused remote
> execution to fail importing templates so it needed to be fixed. The fix made
> openscap and katello broken because they seed job templates. Openscap was
> fixed in time but Katello 3.4 is now uninstallable if people have remote
> execution enabled and we'll need 3.4.1 to fix it.
>
> I think we need to treat some plugins as core. If we can't merge them to core,
> we should make sure, that their changes don't break other plugins. Therefore I
> think we need some job that verifies that as many plugins as possible are
> continuously green. In my opinion, people working on core usually can avoid
> breaking changes if they know about it. If not, they should provide fixes or
> at least guidance for affected plugins.
>
> --
> Marek
>
> On čtvrtek 1. června 2017 18:47:05 CEST Eric D Helms wrote:
>> On Thu, Jun 1, 2017 at 10:18 AM, Tomer Brisker  wrote:
>> > Hello,
>> >
>> > While it is plugin maintainers responsibility to keep their plugins up to
>> > date, one of the most common frustrations they face is "core changed and
>> > broke something AGAIN". I wouldn't want to slow down core's already slow
>> > merge process, but at the same time core contributors should try to
>> > minimize breaking plugins as much as possible - for example, by
>> > deprecating
>> > functions before removing them completely. We could make these tests
>> > optional - i.e. not block merge if plugin tests fail, but just take
>> > advantage of the tests to fix whatever is needed in the plugins sooner
>> > rather then later. I know that in certain cases having katello test run on
>> > every PR let us find and fix issues we wouldn't have known about
>> > otherwise.
>> > Right now contributors (and reviewers) don't have any visibility to
>> > whether a PR will cause problems to other plugins. If breaking is
>> > unavoidable, at least having plugin tests run on PRs will allow giving a
>> > "heads up" to plugin maintainers that they need to fix something, rather
>> > then realizing after the next release that something broke.
>> > During the past couple of months we had several cases of core PRs leading
>> > to broken plugins, which could have been prevented or at least handled
>> > better had we known that a certain change requires changes in plugins.
>>
>> Agree! If I had to sum it up, developer happiness and innovation at speed
>> (core and plugins) is key! One option we've mentioned before that goes
>> directly to Tomer's point is a stable plugin API. We as a community both
>> develop and tell other developers, come, build a plugin. We have a vibrant
>> plugin community. We say plugins are central to the community and project
>> yet CI/CD tells a different story. Do we have a defined contract for
>> plugins? That would solve a lot of the issues IMO. If we had a supported
>> interface (for ALL major use cases of extension, including the UI, since we
>> do have a Foreman::Plugin today) then core could make changes at will as
>> long as the interface passed testing. Any plugin using functionality not in
>> the interface would be SOL but it would be explicitly stated and known.
>>
>> Once we get to installation, running in production mode, scenario testing
>> things get dicier, less stable and with less testing. A good place to
>> invest and bring in other communities (hint: Satlelite QE) to bolster. I'm
>> writing a lot of words without action items which I typically try to avoid.
>> I want to get our testing and matrices there and am working to bring about
>> more time for myself and others to do so. We need to actively invest in
>> this area, make it a priority, and a first class citizen of our community
>> to make an impact.
>>
>> We have wiki pages on creating and releasing plugins. But maybe we need to
>> step back and define what a plugin is to our community. I 

Re: [foreman-dev] Re: Unit test jenkins job with all plugins enabled

[Marek Hulán]

While I agree, one thing that properly defined plugin DSL does not solve is 
the fact, that plugins now depend on each other. Few examples

openscap, katello depends on remote execution
katello, remote execution, ansible, chef, ... depends on foreman tasks
remote execution depends on foreman templates (to some degree)
virt-who-configure depends on katello

Recently, I was lucky to spot that one Katello PR was changing the Hypervisors 
task API while virt-who plugin was subscribed to it, which would cause a 
breakage. We managed to solve that before the PR was merged. Another example 
is that foreman_templates changed import method interface, which caused remote 
execution to fail importing templates so it needed to be fixed. The fix made 
openscap and katello broken because they seed job templates. Openscap was 
fixed in time but Katello 3.4 is now uninstallable if people have remote 
execution enabled and we'll need 3.4.1 to fix it.

I think we need to treat some plugins as core. If we can't merge them to core, 
we should make sure, that their changes don't break other plugins. Therefore I 
think we need some job that verifies that as many plugins as possible are 
continuously green. In my opinion, people working on core usually can avoid 
breaking changes if they know about it. If not, they should provide fixes or 
at least guidance for affected plugins.

--
Marek

On čtvrtek 1. června 2017 18:47:05 CEST Eric D Helms wrote:
> On Thu, Jun 1, 2017 at 10:18 AM, Tomer Brisker  wrote:
> > Hello,
> > 
> > While it is plugin maintainers responsibility to keep their plugins up to
> > date, one of the most common frustrations they face is "core changed and
> > broke something AGAIN". I wouldn't want to slow down core's already slow
> > merge process, but at the same time core contributors should try to
> > minimize breaking plugins as much as possible - for example, by
> > deprecating
> > functions before removing them completely. We could make these tests
> > optional - i.e. not block merge if plugin tests fail, but just take
> > advantage of the tests to fix whatever is needed in the plugins sooner
> > rather then later. I know that in certain cases having katello test run on
> > every PR let us find and fix issues we wouldn't have known about
> > otherwise.
> > Right now contributors (and reviewers) don't have any visibility to
> > whether a PR will cause problems to other plugins. If breaking is
> > unavoidable, at least having plugin tests run on PRs will allow giving a
> > "heads up" to plugin maintainers that they need to fix something, rather
> > then realizing after the next release that something broke.
> > During the past couple of months we had several cases of core PRs leading
> > to broken plugins, which could have been prevented or at least handled
> > better had we known that a certain change requires changes in plugins.
> 
> Agree! If I had to sum it up, developer happiness and innovation at speed
> (core and plugins) is key! One option we've mentioned before that goes
> directly to Tomer's point is a stable plugin API. We as a community both
> develop and tell other developers, come, build a plugin. We have a vibrant
> plugin community. We say plugins are central to the community and project
> yet CI/CD tells a different story. Do we have a defined contract for
> plugins? That would solve a lot of the issues IMO. If we had a supported
> interface (for ALL major use cases of extension, including the UI, since we
> do have a Foreman::Plugin today) then core could make changes at will as
> long as the interface passed testing. Any plugin using functionality not in
> the interface would be SOL but it would be explicitly stated and known.
> 
> Once we get to installation, running in production mode, scenario testing
> things get dicier, less stable and with less testing. A good place to
> invest and bring in other communities (hint: Satlelite QE) to bolster. I'm
> writing a lot of words without action items which I typically try to avoid.
> I want to get our testing and matrices there and am working to bring about
> more time for myself and others to do so. We need to actively invest in
> this area, make it a priority, and a first class citizen of our community
> to make an impact.
> 
> We have wiki pages on creating and releasing plugins. But maybe we need to
> step back and define what a plugin is to our community. I think it would
> also be a fruitful effort to examine the landscape of plugins and see how
> and where they are creating "interfaces" to Foreman.
> 
> 
> Eric
> 
> > As for which plugins to test - I think the best criteria would be to test
> > the 5 or 10 most downloaded plugins or most used plugins according to the
> > community survey.
> > 
> > On Thu, Jun 1, 2017 at 4:50 PM, Lukas Zapletal  wrote:
> >> On Thu, Jun 1, 2017 at 2:55 PM, Eric D Helms 
> >> 
> >> wrote:
> >> > This sounds good in theory, but I'll refresh the 

Re: [foreman-dev] Re: Unit test jenkins job with all plugins enabled

[Eric D Helms]

On Thu, Jun 1, 2017 at 10:18 AM, Tomer Brisker  wrote:

> Hello,
>
> While it is plugin maintainers responsibility to keep their plugins up to
> date, one of the most common frustrations they face is "core changed and
> broke something AGAIN". I wouldn't want to slow down core's already slow
> merge process, but at the same time core contributors should try to
> minimize breaking plugins as much as possible - for example, by deprecating
> functions before removing them completely. We could make these tests
> optional - i.e. not block merge if plugin tests fail, but just take
> advantage of the tests to fix whatever is needed in the plugins sooner
> rather then later. I know that in certain cases having katello test run on
> every PR let us find and fix issues we wouldn't have known about otherwise.
> Right now contributors (and reviewers) don't have any visibility to
> whether a PR will cause problems to other plugins. If breaking is
> unavoidable, at least having plugin tests run on PRs will allow giving a
> "heads up" to plugin maintainers that they need to fix something, rather
> then realizing after the next release that something broke.
> During the past couple of months we had several cases of core PRs leading
> to broken plugins, which could have been prevented or at least handled
> better had we known that a certain change requires changes in plugins.
>

Agree! If I had to sum it up, developer happiness and innovation at speed
(core and plugins) is key! One option we've mentioned before that goes
directly to Tomer's point is a stable plugin API. We as a community both
develop and tell other developers, come, build a plugin. We have a vibrant
plugin community. We say plugins are central to the community and project
yet CI/CD tells a different story. Do we have a defined contract for
plugins? That would solve a lot of the issues IMO. If we had a supported
interface (for ALL major use cases of extension, including the UI, since we
do have a Foreman::Plugin today) then core could make changes at will as
long as the interface passed testing. Any plugin using functionality not in
the interface would be SOL but it would be explicitly stated and known.

Once we get to installation, running in production mode, scenario testing
things get dicier, less stable and with less testing. A good place to
invest and bring in other communities (hint: Satlelite QE) to bolster. I'm
writing a lot of words without action items which I typically try to avoid.
I want to get our testing and matrices there and am working to bring about
more time for myself and others to do so. We need to actively invest in
this area, make it a priority, and a first class citizen of our community
to make an impact.

We have wiki pages on creating and releasing plugins. But maybe we need to
step back and define what a plugin is to our community. I think it would
also be a fruitful effort to examine the landscape of plugins and see how
and where they are creating "interfaces" to Foreman.


Eric


>
> As for which plugins to test - I think the best criteria would be to test
> the 5 or 10 most downloaded plugins or most used plugins according to the
> community survey.
>
> On Thu, Jun 1, 2017 at 4:50 PM, Lukas Zapletal  wrote:
>
>> On Thu, Jun 1, 2017 at 2:55 PM, Eric D Helms 
>> wrote:
>> > This sounds good in theory, but I'll refresh the history of why we did
>> this.
>> > A change would be made to Foreman core, that change would go in and then
>> > propagate out. A developer would update their foreman checkout or the
>> > Katello pipeline would run and break. More developers start updating
>> local
>> > checkouts and now everyone is broken. The Katello PR tests begin to
>> fail and
>> > all PRs are frozen. So now a significant portion of developers are all
>> > trying to figure out what point in time to roll back to, who is going
>> to fix
>> > the breakage and how long it is going to take. 1-5 days later, the
>> issue is
>> > fixed and PRs are able to start back up as well as the nightly pipeline.
>> >
>> > How much developer frustration was created? How much time was lost and
>> > wasted? If we want core to have more freedom of movement then core
>> needs to
>> > evolve to allow plugins more reliability on interfacing.
>>
>> Then let's don't block PRs, I propose nightly job with an email so we
>> are at least aware that things do not work and we can at least wait
>> with RC phase a little bit longer until things settle down, or even
>> block doing another RC until we are all green.
>
>
You lost me here. Blocking PRs is the only way that we reduced developer
frustration. Not blocking means broken code, broken installs, broken
developer environments. Say core merges something that removes a function
Katello relied on. But we did not run Katello unit tests or block at all.
Now a significant chunk of developers can't even boot their application.
Meanwhile, we didn't block PRs due to