Re: Moving ATH tests closer to the code under test

[Basil Crow]

On Wed, May 11, 2022 at 7:32 AM Tim Jacomb  wrote:
> Recently both Jan and myself have hit a number of issues with HtmlUnit not
> supporting modern JavaScript. (You implement it and it works fine in the
> browser but then the PR test builder blows up and you need to re-implement
> it)

jenkinsci/jenkins-test-harness#569 (comment)

describes an idea I had recently:

Another solution would be to teach Can I use/Browserslist/Babel which
> features are (or are not) supported by HtmlUnit/Rhino. Then we could
> transpile our code to something that would run in JTH/HtmlUnit/Rhino with
> minimal modifications. This would be a modest amount of effort and would
> require upstream changes, but it seems relatively achievable in the medium
> term.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjq24pPTz3udwqXP1uZfi8g08EOozxigUb9v89C-C_7o7g%40mail.gmail.com.

Re: Moving ATH tests closer to the code under test

[Tim Jacomb]

I'm in favour of this although I can't dedicate time to it currently.

> Whatever we have, if we end up having ATHs on core PRs to detect
regressions early then we need to be prepared to maintain the ATH
tests/framework or to not accept the changes.

If PRs break the ATH and they are in the same repository of course we would
expect them to be fixed as part of the PR.

> We had a passing build last October, but it did not last (as things where
not run with any associated changes).

We have fairly consistently passing builds, (apart from a number of agent
disconnections requiring re-runs).
Occasionally a core PR will be merged that wasn't run against ATH and
breaks it, or a plugin changes and needs corresponding updates in ATH.

Both of the above issues would be solved by moving the tests to
corresponding plugins and core and then having an aggregator project that
can run tests against all of them like BOM.

If the framework was in core and we replaced existing HtmlUnit tests I
think a lot more would be written.
Recently both Jan and myself have hit a number of issues with HtmlUnit not
supporting modern JavaScript. (You implement it and it works fine in the
browser but then the PR test builder blows up and you need to re-implement
it)

Thanks
Tim

On Wed, 11 May 2022 at 15:11, 'jn...@cloudbees.com' via Jenkins Developers <
jenkinsci-dev@googlegroups.com> wrote:

> Going back to the earlier premise
>
> > It seems to me that a number of problems are caused by ATH tests being
> in a separate repository from the code under test
>
> I would say the biggest problem is not so much that - rather that we do
> not run enough of these tests for the code that is affected, early enough,
> nor is there probably a good set of well written smoke test candidates.
>
> Moving the code is one possible solution to the first issue, but it also
> has other problems.  For example If we update the PageObjects you then need
> to update N repos (or just not bother) to account for the change in API.
> (granted changes in API are hopefully minimal, but that is not always the
> case, esp wrt things like table-2-div).
>
> I discussed with directly with Mark Waite, but to me we need some new
> tests that better cover what we consider to be "critical functionality",
> (ie an Acceptance test), what we tend to have today is a bunch of limited
> tests where things are testing some things in isolation.
>
> I would love to see a test that was run on a core PR that at least created
> a credentials (in different scopes, stores), an agent a a pipeline.  And
> check that the pipeline can checkout from a Git Repo  (github would be
> nice, but attempting to minimize network operations and hence flakes and
> also brings security issues into play...) and build some PRs and a main
> branch.
> And something similar with a FreeStyleJob.  (include in that Junit
> reporting).  Currently the smoke tests for core PRs are too fine grained to
> me.
>
> I think we could make some quick wins with the following form of attack
>
> *  extend buildPlugin so that you can specify a list of ATH tests to run
> (currently most of the tests are in the form of plugins.Test
> * introduce a few new tests like discussed above that are run on Jenkins
> PRs.
>
> Whatever we have, if we end up having ATHs on core PRs to detect
> regressions early then we need to be prepared to maintain the ATH
> tests/framework or to not accept the changes.  The maintenance has mostly
> fallen to a few individuals like Tim, Vincent and myself.  We had a passing
> build last October, but it did not last (as things where not run with any
> associated changes).
>
> I am all in favour of improving the current situation however that happens
> and am happy to see this gaining traction.
>
> /James
>
> On Wednesday, April 27, 2022 at 7:33:44 PM UTC+1 ullrich...@gmail.com
> wrote:
>
>>
>> > Am 26.04.2022 um 18:28 schrieb Basil Crow :
>> >
>> > On Tue, Apr 26, 2022 at 7:45 AM Ullrich Hafner 
>> wrote:
>> >
>> >> Those tests will run in a special docker container with a headless
>> browser (chrome or firefox).
>> >
>> > Ideally, I think we would want to support both Dockerized and
>> > non-Dockerized builds if possible.
>>
>> You can run the build locally without Docker. Docker is just used in CI
>> (GitHub Action or Jenkins). I am locally using a different Maven profile
>> and run the tests without Docker in Firefox or Chrome. You can even run it
>> from IntelliJ and set breakpoints.
>>
>> >
>> >> […] run them as real system tests using a real browser and a real
>> Jenkins instance and not as part of our integration tests with a fake UI
>> (HTMLUnit) and a fake Jenkins (started via JTH) […]
>> >
>> > Regarding the use of a "real Jenkins instance," I wonder how the ATH
>> > implementation compares to RealJenkinsRule, which is at a high level
>> > doing something very similar; namely, starting Jenkins in a separate
>> > Java process and tearing it down at the end of the test. If the
>> > RealJenkinsRule 

Re: Moving ATH tests closer to the code under test

['jn...@cloudbees.com' via Jenkins Developers]

Going back to the earlier premise

> It seems to me that a number of problems are caused by ATH tests being
in a separate repository from the code under test

I would say the biggest problem is not so much that - rather that we do not 
run enough of these tests for the code that is affected, early enough, nor 
is there probably a good set of well written smoke test candidates.

Moving the code is one possible solution to the first issue, but it also 
has other problems.  For example If we update the PageObjects you then need 
to update N repos (or just not bother) to account for the change in API.  
(granted changes in API are hopefully minimal, but that is not always the 
case, esp wrt things like table-2-div).

I discussed with directly with Mark Waite, but to me we need some new tests 
that better cover what we consider to be "critical functionality", (ie an 
Acceptance test), what we tend to have today is a bunch of limited tests 
where things are testing some things in isolation.

I would love to see a test that was run on a core PR that at least created 
a credentials (in different scopes, stores), an agent a a pipeline.  And 
check that the pipeline can checkout from a Git Repo  (github would be 
nice, but attempting to minimize network operations and hence flakes and 
also brings security issues into play...) and build some PRs and a main 
branch.
And something similar with a FreeStyleJob.  (include in that Junit 
reporting).  Currently the smoke tests for core PRs are too fine grained to 
me.

I think we could make some quick wins with the following form of attack

*  extend buildPlugin so that you can specify a list of ATH tests to run 
(currently most of the tests are in the form of plugins.Test
* introduce a few new tests like discussed above that are run on Jenkins 
PRs.

Whatever we have, if we end up having ATHs on core PRs to detect 
regressions early then we need to be prepared to maintain the ATH 
tests/framework or to not accept the changes.  The maintenance has mostly 
fallen to a few individuals like Tim, Vincent and myself.  We had a passing 
build last October, but it did not last (as things where not run with any 
associated changes).

I am all in favour of improving the current situation however that happens 
and am happy to see this gaining traction.

/James

On Wednesday, April 27, 2022 at 7:33:44 PM UTC+1 ullrich...@gmail.com wrote:

>
> > Am 26.04.2022 um 18:28 schrieb Basil Crow : 
> > 
> > On Tue, Apr 26, 2022 at 7:45 AM Ullrich Hafner  
> wrote: 
> > 
> >> Those tests will run in a special docker container with a headless 
> browser (chrome or firefox). 
> > 
> > Ideally, I think we would want to support both Dockerized and 
> > non-Dockerized builds if possible. 
>
> You can run the build locally without Docker. Docker is just used in CI 
> (GitHub Action or Jenkins). I am locally using a different Maven profile 
> and run the tests without Docker in Firefox or Chrome. You can even run it 
> from IntelliJ and set breakpoints. 
>
> > 
> >> […] run them as real system tests using a real browser and a real 
> Jenkins instance and not as part of our integration tests with a fake UI 
> (HTMLUnit) and a fake Jenkins (started via JTH) […] 
> > 
> > Regarding the use of a "real Jenkins instance," I wonder how the ATH 
> > implementation compares to RealJenkinsRule, which is at a high level 
> > doing something very similar; namely, starting Jenkins in a separate 
> > Java process and tearing it down at the end of the test. If the 
> > RealJenkinsRule implementation is more efficient while still remaining 
> > compatible with the goal of frontend testing with a headless browser, 
> > it may be worth looking into unifying the two implementations in order 
> > to reduce maintenance and/or increase performance. 
> > 
> >> not all plugins have UI tests yet, so there is an initial ramp up time 
> required 
> > 
> > I think if we build it they will come. 
> > 
> >> […] UI tests are more fragile 
> > 
> > Has this ever not been the case in the decades you and I have been 
> > programming? I guess the benefit of this idea depends on _how_ 
> > fragile. If the fragility is occasional and a retry chases the problem 
> > away, that could be tolerable. On the other hand, if the fragility is 
> > frequent and requires one or more retries, this could undermine the 
> > idea. 
> > 
> >> build execution times of PRs will increase since the initialization is 
> slow 
> > 
> > I wonder if some targeted optimization couldn't be done through the 
> > use of techniques such as the ones used in RealJenkinsRule. 
> > 
> > Basil 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Jenkins Developers" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to jenkinsci-de...@googlegroups.com. 
> > To view this discussion on the web visit 
> 

Re: Moving ATH tests closer to the code under test

[Ullrich Hafner]

> Am 26.04.2022 um 18:28 schrieb Basil Crow :
> 
> On Tue, Apr 26, 2022 at 7:45 AM Ullrich Hafner  
> wrote:
> 
>> Those tests will run in a special docker container with a headless browser 
>> (chrome or firefox).
> 
> Ideally, I think we would want to support both Dockerized and
> non-Dockerized builds if possible. 

You can run the build locally without Docker. Docker is just used in CI (GitHub 
Action or Jenkins). I am locally using a different Maven profile and run the 
tests without Docker in Firefox or Chrome. You can even run it from IntelliJ 
and set breakpoints.

> 
>> […] run them as real system tests using a real browser and a real Jenkins 
>> instance and not as part of our integration tests with a fake UI (HTMLUnit) 
>> and a fake Jenkins (started via JTH) […]
> 
> Regarding the use of a "real Jenkins instance," I wonder how the ATH
> implementation compares to RealJenkinsRule, which is at a high level
> doing something very similar; namely, starting Jenkins in a separate
> Java process and tearing it down at the end of the test. If the
> RealJenkinsRule implementation is more efficient while still remaining
> compatible with the goal of frontend testing with a headless browser,
> it may be worth looking into unifying the two implementations in order
> to reduce maintenance and/or increase performance.
> 
>> not all plugins have UI tests yet, so there is an initial ramp up time 
>> required
> 
> I think if we build it they will come.
> 
>> […] UI tests are more fragile
> 
> Has this ever not been the case in the decades you and I have been
> programming? I guess the benefit of this idea depends on _how_
> fragile. If the fragility is occasional and a retry chases the problem
> away, that could be tolerable. On the other hand, if the fragility is
> frequent and requires one or more retries, this could undermine the
> idea.
> 
>> build execution times of PRs will increase since the initialization is slow
> 
> I wonder if some targeted optimization couldn't be done through the
> use of techniques such as the ones used in RealJenkinsRule.
> 
> Basil
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjphex3uXLfjsAfCpEaFvMNpXgBocgnX5tHaSuqTV1iZ7w%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/644842DF-A9B1-4D27-81F7-AE5598E3325B%40gmail.com.

Re: Moving ATH tests closer to the code under test

[Basil Crow]

On Wed, Apr 27, 2022 at 9:06 AM Ullrich Hafner  wrote:
>
> This will depend on the type of the plugins. In my analysis plugins my tests 
> start by creating a freestyle job or pipeline. There is no previous setup 
> required. I don't think that this can be speed up with job-dsl or something 
> different.

The point is that creating the project isn't the "interesting" part of
your test. It could be done with white-box JTH/RealJenkinsRule code
rather than black-box ATH/Selenium code and you presumably wouldn't
care, as long as the "interesting" part of your test (verifying that
analysis results are present, clicking into graphs in the UI, etc) was
tested by black-box ATH/Selenium code. And as long the parts that are
"uninteresting" in the context of your analysis plugins are deemed
"interesting" by some other plugin (or core) and tested by black-box
ATH/Selenium logic there, then we can maintain the same level of
coverage across the whole Jenkins project at a macro level.

In this example, we could have a black-box ATH/Selenium test that
creates a Freestyle job in core and a black-box ATH/Selenium test that
creates a Pipeline job in Workflow: Job. Functional testing for
creating those job types would then be covered sufficiently, and in
your analysis plugins you could take the shortcut of using white-box
JTH/RealJenkinsRule logic to create the job efficiently (and
reliably!) before proceeding to the "interesting" part of your test
with black-box ATH/Selenium logic.

The point is that efficiency and reliability can be improved by
removing duplicate effort. Expensive and flaky black-box ATH/Selenium
logic to create a Freestyle job doesn't need to run thousands of
times. It only needs to run a handful of times in key repositories,
and the other repositories can assume that it would work and test on
top of that, a form of proof by induction.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjp%2BnU5dpPB3j6cd_q3gz0KSuZ%2B9Md%3DVBX3ypJbq%2B%3D5%3D2g%40mail.gmail.com.

Re: Moving ATH tests closer to the code under test

[Ullrich Hafner]

> Am 27.04.2022 um 12:37 schrieb 'Antonio Muñiz' via Jenkins Developers 
> :
> 
> I was working/planning with Oliver for some time on this. One of the 
> identified downsides of this approach is that the Jenkins UI does often 
> evolve in a backward incompatible way (ie. table to divs change, plugin 
> manager recent changes, etc), all those potential breaking changes would not 
> be caught by acceptance tests running at plugin build/release time (unless 
> the plugin maintainer updates the `jenkins.version` to the latest very often).

When you run the ATH tests in a plugin then you run the tests agains the latest 
Jenkins version, not the baseline version defined in the plugin. I’m using 
Dependabot to make sure that this version is up to date. (It is still too late 
since the Jenkins release is already published, but at least you will see it 
one day after the release)

> The current ATH approach runs all acceptance tests against a specific Jenkins 
> version, so possible regressions are caught at Jenkins core release time, as 
> desired.
> The alternative would be to define the plugin build/release pipeline in a way 
> that ATHs are run against a dynamic Jenkins version (could be the latest 
> weekly at the time of the build), but that leads to unreproducible builds, 
> which is IMO quite bad. And it does not address the issue of checking for 
> regressions caused on plugins at core release time (unless some complicated 
> pipeline is put in place to run ATHs from some key plugins as part of the 
> Jenkins core build).
> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/C060C521-3E3A-4186-84EC-673A779FAFB2%40gmail.com.

Re: Moving ATH tests closer to the code under test

[Ullrich Hafner]

>> Although ATH uses RESTish endpoints for a few purposes, for the most part 
>> the test setup (not just the actual assertions) uses the browser. In all 
>> cases the whole test run is “black-box”. This has its appeal (stronger 
>> coverage) but GUI setup makes tests much slower and either REST or GUI setup 
>> can also make it a lot more awkward to write tests than when using 
>> `[Real]JenkinsRule`, which are “white-box” and let you quickly set up an 
>> environment and run some assertions at the Java level. JTH also lets you 
>> write test-only extensions, which ATH does not.
> 
> I see; very interesting. I wonder how many parts of the test setup
> process are common to many tests/plugins versus unique to each
> test/plugin. For example, I could imagine that things like going
> through the setup wizard, creating a freestyle project, or adding an
> agent are common to many tests (and therefore pointless from the
> perspective of test coverage to repeat them multiple times), while
> other parts of the test setup process, like configuring a specific
> plugin, might be unique to each test/plugin. If true, and if the test
> time is dominated by the parts of the test setup process that are
> common to many tests/plugins, perhaps we could, for each portion of
> the test setup process that is common to many tests/plugins, run that
> portion with the GUI/REST API in one "primary" repository (e.g., core,
> Workflow: Job, etc) and (without loss of test coverage at a macro
> level) run that portion with JCasC/REST or even Java/JTH in the
> "secondary" repositories as a way of speeding up and de-flaking the
> setup process to get to the unique/interesting part.

This will depend on the type of the plugins. In my analysis plugins my tests 
start by creating a freestyle job or pipeline. There is no previous setup 
required. I don't think that this can be speed up with job-dsl or something 
different.  

> 
> In other words, I wonder if a middle ground between test coverage and
> performance couldn't be found. Or maybe I am hopelessly naive and this
> is an exercise in futility in the long term. I will discuss this with
> my friend who has experience in this domain.
> 
>> ATH installs plugins, and selects plugin versions, using user-mode tools. 
>> `[Real]JenkinsRule` follow the Maven test classpath. Possibly ATH could be 
>> given an option to define a mock UC based on a test classpath.
> 
> Yes, a great example of the "middle ground" approach from the previous
> paragraph!

This is already possible. In my UI tests I am starting the Jenkins under test 
with the same set of plugins that I am using in the integration tests (JTH). 
These plugins are just copied from the target folder of the plugin maven module 
to the target folder of the ui module and the JUT starts with them without 
installing them.

> 
>> Also whether the fragility is in the actual assertions, or test setup (point 
>> #1 above).
> 
> I think your point is that fragility in the test setup would undermine
> the idea at its core, while fragility in the actual assertions could
> be tolerated on a case-by-case basis in individual test suites. If so,
> I concur, and I think that finding a way to mitigate the fragility of
> test setup by factoring out that logic as described in the preceding
> paragraphs could be one solution.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjpmQLNRwhJSGYc3wT28P32gHvt2aF_CJtkDtX47iwBGzw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CCF7B65B-3986-4C78-B66C-1417C01264F5%40gmail.com.

Re: Moving ATH tests closer to the code under test

[Basil Crow]

On Wed, Apr 27, 2022 at 3:37 AM 'Antonio Muñiz' via Jenkins Developers
 wrote:
> unless some complicated pipeline is put in place to run ATHs from some key 
> plugins as part of the Jenkins core build

Yes, implied in this proposal was running the plugin BOM/PCT tests
(which, as of this proposal, would also include ATH tests) against
core PRs that are ready for merge prior to merging and releasing them
as described in https://github.com/jenkinsci/bom/issues/758. The
general point is to shift testing as far leftward as possible given
budgetary constraints.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjpCYXqccxCu6SuKiKQmU%3DqcvgLQ%2BGtsAk3mNNbcoonr5g%40mail.gmail.com.

Re: Moving ATH tests closer to the code under test

['Antonio Muñiz' via Jenkins Developers]

I was working/planning with Oliver for some time on this. One of the
identified downsides of this approach is that the Jenkins UI does often
evolve in a backward incompatible way (ie. table to divs change, plugin
manager recent changes, etc), all those potential breaking changes would
not be caught by acceptance tests running at plugin build/release time
(unless the plugin maintainer updates the `jenkins.version` to the latest
very often). The current ATH approach runs all acceptance tests against a
specific Jenkins version, so possible regressions are caught at Jenkins
core release time, as desired.
The alternative would be to define the plugin build/release pipeline in a
way that ATHs are run against a dynamic Jenkins version (could be the
latest weekly at the time of the build), but that leads to unreproducible
builds, which is IMO quite bad. And it does not address the issue of
checking for regressions caused on plugins at core release time (unless
some complicated pipeline is put in place to run ATHs from some key plugins
as part of the Jenkins core build).

On Tue, 26 Apr 2022 at 20:57, Basil Crow  wrote:

> On Tue, Apr 26, 2022 at 11:12 AM 'Jesse Glick' via Jenkins Developers
>  wrote:
>
> > `acceptance-test-harness` has a bunch of dependencies some of which
> clash with those in Jenkins core or some plugins, so you would need to
> either shade them all, or otherwise somehow ensure the ATH dependency
> trails are given minimum priority.
>
> I see; thank you. I suppose the same problem exists in JTH, with
> shading being the solution, so I see no reason why the same couldn't
> be done for ATH.
>
> > Although ATH uses RESTish endpoints for a few purposes, for the most
> part the test setup (not just the actual assertions) uses the browser. In
> all cases the whole test run is “black-box”. This has its appeal (stronger
> coverage) but GUI setup makes tests much slower and either REST or GUI
> setup can also make it a lot more awkward to write tests than when using
> `[Real]JenkinsRule`, which are “white-box” and let you quickly set up an
> environment and run some assertions at the Java level. JTH also lets you
> write test-only extensions, which ATH does not.
>
> I see; very interesting. I wonder how many parts of the test setup
> process are common to many tests/plugins versus unique to each
> test/plugin. For example, I could imagine that things like going
> through the setup wizard, creating a freestyle project, or adding an
> agent are common to many tests (and therefore pointless from the
> perspective of test coverage to repeat them multiple times), while
> other parts of the test setup process, like configuring a specific
> plugin, might be unique to each test/plugin. If true, and if the test
> time is dominated by the parts of the test setup process that are
> common to many tests/plugins, perhaps we could, for each portion of
> the test setup process that is common to many tests/plugins, run that
> portion with the GUI/REST API in one "primary" repository (e.g., core,
> Workflow: Job, etc) and (without loss of test coverage at a macro
> level) run that portion with JCasC/REST or even Java/JTH in the
> "secondary" repositories as a way of speeding up and de-flaking the
> setup process to get to the unique/interesting part.
>
> In other words, I wonder if a middle ground between test coverage and
> performance couldn't be found. Or maybe I am hopelessly naive and this
> is an exercise in futility in the long term. I will discuss this with
> my friend who has experience in this domain.
>
> > ATH installs plugins, and selects plugin versions, using user-mode
> tools. `[Real]JenkinsRule` follow the Maven test classpath. Possibly ATH
> could be given an option to define a mock UC based on a test classpath.
>
> Yes, a great example of the "middle ground" approach from the previous
> paragraph!
>
> > Also whether the fragility is in the actual assertions, or test setup
> (point #1 above).
>
> I think your point is that fragility in the test setup would undermine
> the idea at its core, while fragility in the actual assertions could
> be tolerated on a case-by-case basis in individual test suites. If so,
> I concur, and I think that finding a way to mitigate the fragility of
> test setup by factoring out that logic as described in the preceding
> paragraphs could be one solution.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjpmQLNRwhJSGYc3wT28P32gHvt2aF_CJtkDtX47iwBGzw%40mail.gmail.com
> .
>


-- 
Antonio Muñiz
Human, Engineer
CloudBees, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails 

Re: Moving ATH tests closer to the code under test

[Basil Crow]

On Tue, Apr 26, 2022 at 11:12 AM 'Jesse Glick' via Jenkins Developers
 wrote:

> `acceptance-test-harness` has a bunch of dependencies some of which clash 
> with those in Jenkins core or some plugins, so you would need to either shade 
> them all, or otherwise somehow ensure the ATH dependency trails are given 
> minimum priority.

I see; thank you. I suppose the same problem exists in JTH, with
shading being the solution, so I see no reason why the same couldn't
be done for ATH.

> Although ATH uses RESTish endpoints for a few purposes, for the most part the 
> test setup (not just the actual assertions) uses the browser. In all cases 
> the whole test run is “black-box”. This has its appeal (stronger coverage) 
> but GUI setup makes tests much slower and either REST or GUI setup can also 
> make it a lot more awkward to write tests than when using 
> `[Real]JenkinsRule`, which are “white-box” and let you quickly set up an 
> environment and run some assertions at the Java level. JTH also lets you 
> write test-only extensions, which ATH does not.

I see; very interesting. I wonder how many parts of the test setup
process are common to many tests/plugins versus unique to each
test/plugin. For example, I could imagine that things like going
through the setup wizard, creating a freestyle project, or adding an
agent are common to many tests (and therefore pointless from the
perspective of test coverage to repeat them multiple times), while
other parts of the test setup process, like configuring a specific
plugin, might be unique to each test/plugin. If true, and if the test
time is dominated by the parts of the test setup process that are
common to many tests/plugins, perhaps we could, for each portion of
the test setup process that is common to many tests/plugins, run that
portion with the GUI/REST API in one "primary" repository (e.g., core,
Workflow: Job, etc) and (without loss of test coverage at a macro
level) run that portion with JCasC/REST or even Java/JTH in the
"secondary" repositories as a way of speeding up and de-flaking the
setup process to get to the unique/interesting part.

In other words, I wonder if a middle ground between test coverage and
performance couldn't be found. Or maybe I am hopelessly naive and this
is an exercise in futility in the long term. I will discuss this with
my friend who has experience in this domain.

> ATH installs plugins, and selects plugin versions, using user-mode tools. 
> `[Real]JenkinsRule` follow the Maven test classpath. Possibly ATH could be 
> given an option to define a mock UC based on a test classpath.

Yes, a great example of the "middle ground" approach from the previous
paragraph!

> Also whether the fragility is in the actual assertions, or test setup (point 
> #1 above).

I think your point is that fragility in the test setup would undermine
the idea at its core, while fragility in the actual assertions could
be tolerated on a case-by-case basis in individual test suites. If so,
I concur, and I think that finding a way to mitigate the fragility of
test setup by factoring out that logic as described in the preceding
paragraphs could be one solution.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjpmQLNRwhJSGYc3wT28P32gHvt2aF_CJtkDtX47iwBGzw%40mail.gmail.com.

Re: Moving ATH tests closer to the code under test

['Jesse Glick' via Jenkins Developers]

On Tue, Apr 26, 2022 at 12:28 PM Basil Crow  wrote:

> I wonder if we could design a system such that for the common case only a
> single Maven project is needed


Sounds tricky. `acceptance-test-harness` has a bunch of dependencies some
of which clash with those in Jenkins core or some plugins, so you would
need to either shade them all, or otherwise somehow ensure the ATH
dependency trails are given minimum priority.

I wonder how the ATH implementation compares to RealJenkinsRule


You could in principle use `RealJenkinsRule` with some GUI testing library
other than HtmlUnit that better supports exotic JavaScript. If you wanted
to reuse the rich Selenium page object library in ATH you would need to
factor that out into a standalone library.

There are other significant differences:

   - Although ATH uses RESTish endpoints for a few purposes, for the most
   part the test *setup* (not just the actual assertions) uses the browser.
   In all cases the whole test run is “black-box”. This has its appeal
   (stronger coverage) but GUI setup makes tests much slower and either REST
   or GUI setup can also make it a lot more awkward to write tests than when
   using `[Real]JenkinsRule`, which are “white-box” and let you quickly set up
   an environment and run some assertions at the Java level. JTH also lets you
   write test-only extensions, which ATH does not.
   - ATH installs plugins, and selects plugin versions, using user-mode
   tools. `[Real]JenkinsRule` follow the Maven test classpath. Possibly ATH
   could be given an option to define a mock UC based on a test classpath.
   - ATH uses Guice for configuration. JTH does not.
   - ATH uses `docker-fixtures` for containerized tests. JTH can use that
   also, but Testcontainers is recommended instead.



> the benefit of this idea depends on _how_ fragile.


Also whether the fragility is in the actual assertions, or test setup
(point #1 above).

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2HJN1KKt0D5c%2BYsPAhMMvw5ThwGTbqxBLGjEaEckRUwA%40mail.gmail.com.

Re: Moving ATH tests closer to the code under test

[Basil Crow]

On Tue, Apr 26, 2022 at 7:45 AM Ullrich Hafner  wrote:
> Thanks Basil for bringing this topic to attention!

You are welcome!

> […] I already converted several plugins in the way you are suggesting […]

I am pleased that this approach has been working well in your plugins.
Assuming that the idea itself is desirable (and by all means, let me
know if you think it is not!), let me focus on some practical concerns
around the feasibility of adopting such an approach throughout the
core and plugin ecosystem. One of the strengths of this ecosystem is
that we expose a rich and polished framework to plugin developers that
works out-of-the-box for most use cases. Adding this type of testing
to the toolchain while maintaining existing levels of approachability
could be a challenge, so there are a few points I want to discuss.

> […] we need a maven sub-module for the plugin and one for the corresponding 
> ui-tests.

Ideally, I think we would avoid the use of multi-module Maven projects
if possible. The vast majority of Jenkins plugins are single-module
Maven projects, and in my opinion the simplicity and low maintenance
associated with this is desirable. I wonder if we could design a
system such that for the common case only a single Maven project is
needed, while still supporting multi-module Maven projects for more
complex use cases. I am not sure if your existing multi-module Maven
project is due to technical limitations or for
structural/organizational reasons.

> Those tests will run in a special docker container with a headless browser 
> (chrome or firefox).

Ideally, I think we would want to support both Dockerized and
non-Dockerized builds if possible. While most developers today are
running Docker, it adds some complexity to the process of debugging
local builds and adds some infrastructure requirements for CI builds.
I think decreasing the number of requirements and the amount of
complexity would make it easier for such a system to gain widespread
adoption. Also, we need to think about how such changes would impact
existing BOM/PCT builds. If we retain a single Maven module and avoid
the requirement for a Docker daemon, then I think few if any changes
would be needed here. But if we introduce additional complexity, there
might be further infrastructure changes needed in BOM/PCT.

> […] run them as real system tests using a real browser and a real Jenkins 
> instance and not as part of our integration tests with a fake UI (HTMLUnit) 
> and a fake Jenkins (started via JTH) […]

Regarding the use of a "real Jenkins instance," I wonder how the ATH
implementation compares to RealJenkinsRule, which is at a high level
doing something very similar; namely, starting Jenkins in a separate
Java process and tearing it down at the end of the test. If the
RealJenkinsRule implementation is more efficient while still remaining
compatible with the goal of frontend testing with a headless browser,
it may be worth looking into unifying the two implementations in order
to reduce maintenance and/or increase performance.

> not all plugins have UI tests yet, so there is an initial ramp up time 
> required

I think if we build it they will come.

> […] UI tests are more fragile

Has this ever not been the case in the decades you and I have been
programming? I guess the benefit of this idea depends on _how_
fragile. If the fragility is occasional and a retry chases the problem
away, that could be tolerable. On the other hand, if the fragility is
frequent and requires one or more retries, this could undermine the
idea.

> build execution times of PRs will increase since the initialization is slow

I wonder if some targeted optimization couldn't be done through the
use of techniques such as the ones used in RealJenkinsRule.

Basil

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjphex3uXLfjsAfCpEaFvMNpXgBocgnX5tHaSuqTV1iZ7w%40mail.gmail.com.

Re: Moving ATH tests closer to the code under test

[Ullrich Hafner]

Thanks Basil for bringing this topic to attention! 

I am all in for this new approach! I think when we spend time to write UI tests 
(which are expensive to write and maintain, and slow to run, see test pyramid 
of Mike Cohen) then we should consider to run them as real system tests using a 
real browser and a real Jenkins instance and not as part of our integration 
tests with a fake UI (HTMLUnit) and a fake Jenkins (started via JTH). 

Since I already converted several plugins in the way you are suggesting I can 
give some more insights about the things that need to be done, the advantages, 
and disadvantages. I can only speak for plugins but in theory the same process 
should be possible for core as well. 

First of all, the credits for this approach go to Oliver Gondža! He created the 
building blocks in the acceptance test harness and his own plugins. If we 
follow this approach, a plugin will have three different types of tests:
1) Unit tests that test classes in isolation.
2) Integration tests that test one or more classes together in a running 
Jenkins instance (started by JTH). These tests will solely work on Jenkins' 
object model and will not use the browser HTML output anymore.
3) UI tests as end-to-end tests for one or more scenarios of the plugin use 
cases.

These tests will then be started using maven-surefire (1) and maven-failsafe 
(2) in our normal pipelines (buildPlugin() or GitHub Actions). Those results 
will visible in the usual way in a pull request. The UI tests (3) need to be 
started as a separate process since they work on the created artifacts. I’m 
using GitHub Actions in my plugins to start the individual UI tests, but it 
should be possible to run on Jenkins as well. Those tests will run in a special 
docker container with a headless browser (chrome or firefox).

All three types of tests are started for every pull request. So if a committer 
breaks something in the UI it will be visible as a PR check failure. Even if 
Jenkins core changes from 1.n to 1.(n+1), a Dependabot PR will be triggered and 
evaluated. 

If you want to use that approach in your plugin you need the following steps:
1) Change the folder structure of a plugin: we need a maven sub-module for the 
plugin and one for the corresponding ui-tests. Basically, all existing files 
need to be moved below the plugin folder.
2) Create a new ui-test project for the plugin.
3) Move existing Selenium page objects and test cases from the ATH to the 
plugin.  

Then you will have the previous ATH tests as part of your plugin. Here are some 
working examples:
- a feature PR in the Warnings Plugin 
(https://github.com/jenkinsci/warnings-ng-plugin/pull/1206 
)
- Jenkins upgrade PR in the Code Coverage Plugin 
(https://github.com/jenkinsci/code-coverage-api-plugin/pull/377 
)

Here are two work-in-progress PRs:
- JUnit Plugin (https://github.com/jenkinsci/junit-plugin/pull/355 
 or 
https://github.com/jenkinsci/junit-plugin/pull/342 
)
- PIT Plugin (https://github.com/jenkinsci/pitmutation-plugin/pull/129 
)

So I think that it something that each plugin author can do without much effort 
in each plugin. 

So as a summary the advantages:
- UI tests work with a real browser (and do not break due to HTMLUnit 
incompatibilities)
- integration tests are fast and work with assertions on the Jenkins object 
model. 
- when you do not use HTMLUnit anymore you will also find better ways to test 
you Java side models. This will lead to a better Java model! The UI layer 
should be as thin as possible and the logic will be part of Java that can be 
tested with Unit tests in Java.
- both types of tests are evaluated for each pull request

The disadvantages:
- not all plugins have UI tests yet, so there is an initial ramp up time 
required
- ATH based UI tests are more fragile
- build execution times of PRs will increase since the initialization is slow


> Am 25.04.2022 um 21:19 schrieb Basil Crow :
> 
> Hi all,
> 
> It seems to me that a number of problems are caused by ATH tests being
> in a separate repository from the code under test. When regressions
> occur, they are not discovered until much later. This causes pain for
> those who investigate ATH test failures. If the ATH tests were moved
> to the same repository as the code under test (i.e., core or plugins),
> then these issues would be caught prior to the problematic change
> being merged and released. This would be a win for those who
> investigate ATH failures.
> 
> Related is the fact that ATH provides a more realistic execution
> environment than JTH. ATH supports modern JavaScript frameworks, but
> JTH does not. Being able to easily write ATH tests in core and plugins
> would be a win for 

Moving ATH tests closer to the code under test

[Basil Crow]

Hi all,

It seems to me that a number of problems are caused by ATH tests being
in a separate repository from the code under test. When regressions
occur, they are not discovered until much later. This causes pain for
those who investigate ATH test failures. If the ATH tests were moved
to the same repository as the code under test (i.e., core or plugins),
then these issues would be caught prior to the problematic change
being merged and released. This would be a win for those who
investigate ATH failures.

Related is the fact that ATH provides a more realistic execution
environment than JTH. ATH supports modern JavaScript frameworks, but
JTH does not. Being able to easily write ATH tests in core and plugins
would be a win for those who make frontend changes. New features could
be delivered with associated test automation at the time of feature
delivery.

I think moving the ATH tests to the same repository as the code under
test would be a win on both fronts. ATH and JTH would then both be
test _harnesses_ (as their names imply) but not _collections_ of
tests: those would live in the same repository as the code under test,
and passing them would be a prerequisite for merge and release.

Integrating such a system into the toolchain would be a significant
project. Providing first-class support for ATH tests in core and
plugins in the context of local builds, plugin CI builds, and BOM/PCT
builds requires a significant amount of work and likely carries
infrastructure prerequisites. Nevertheless, I think such a project
would be of large benefit. It would increase our confidence when
making UI changes, and it would increase our confidence that BOM bumps
do not contain regressions. These benefits would apply to multiple
parties across the Jenkins project.

Note that this is a proposal for a project and a request for
discussion, but I am explicitly not volunteering to implement the
solution (at least not by myself).

Basil

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjoGK5XdfP38zE1WPH-pcVefRrOc7ndvWqApFMzEGEE0zg%40mail.gmail.com.