Hi, That definitely sounds interesting! See my replies inline:
On Thu, 29 Oct 2015 14:26:59 +0100, Aditya Nisal
<[email protected]> wrote:
Hello Ulli!
Thank you for your valuable feedback! Next time onwards I would
definitely
ask such questions in the dev mailing list. I did post my questions in
the
#jenkins IRC channel but decided to mail directly the repository
contributors, since I did not get any response there.
My thesis aims to analyze the robustness and stability of selenium tests
over the version history of Jenkins. Thus, I am going to take 3 major
versions of Jenkins (1.4xx,1.5xx,1.6xx) and their respective minor
releases, and execute the acceptance test suite against them. However I
was
wondering if after a major version is released, if the test-cases need a
significant modification in order to maintain the same coverage/passed
tests. Hence I would like to know if there are any such points (perhaps
identified by a particular tag) when the tests underwent a major
overhaul?
Some remarks:
- Make sure to pick LTS releases as those tend to be lot more stable than
weekly ones. Comparing weekly with LTS can give seriously biased results.
- Note that there are some test annotated with @Since("X") that does not
run with Jenkins version older than X as covered use-case was not
supported yet. Later versions can then produce more successful tests
without improving the stability.
- It can be even more tricky with bundled plugin version constrains
@WithPlugins("Y@X")
- Make sure to run the suite with docker daemon on not to skip docker
based tests.
- There are no major version of Jenkins (in traditional sense), though the
core/plugin evolution breaks ATH occasionally.
- Usually new tests are developed with "recent" versions in mind and
whoever needs to run ATH against older version contributes his/her time to
keep ATH running on that version.
- Last version I used ATH against was 1.480, I am afraid it would take a
significant effort for you to make it running for anything older.
- Note that such "incompatibilities" between ATH and Jenkins will be
more common for older releases. Again, newer versions can appear more
stable than old ones because of that.
- Tests in ATH are rather "sparse" meaning it support only a fracture of
Jenkins use-cases (compared to jenkins-test-harness for instance). I am
not sure how reliable the results will be.
I am also planning to publish the results after my experiment and inform
about them in the irc channel, in case someone would like to use a
feedback.
jenkinsci-dev would be definitely interested in that!
On 29 October 2015 at 13:34, Ullrich Hafner <[email protected]>
wrote:
Hi Adi,
first of all: such questions should be asked in the dev mailing list so
others can participate (or more importantly see the answers later on).
Am 29.10.2015 um 11:20 schrieb Aditya Nisal
<[email protected]>:
Hello all!
My name is Adi and I am a MSc Computer Science student from Saarland
University, Germany. As a part of my master's thesis I am analyzing
Jenkins
Selenium tests (https://github.com/jenkinsci/acceptance-test-harness/).
I had a question regarding the release history of this test suite. As
per
the releases on the github repository, there are 16 releases (tags),
however, I did not understand if there is any mapping /relationship
between
Jenkins' releases and the acceptance tests' releases. I investigated the
pom.xml of all releases, but the
parent pom always points to Jenkins version 1.32.
<parent>
<groupId>org.jenkins-ci</groupId>
<artifactId>jenkins</artifactId>
<version>1.32</version>
</parent>
This dependency is only there to import some basic properties which are
used both by Jenkins and the test suite. Actually there is (almost) no
relationship from the test suite to a Jenkins release. I.e. the tests
should run for several releases of Jenkins. And in fact we already run
the
same suite for Jenkins LTS and DEV. (Internally we specify for test
cases
for which release a test case is created for.)
Moreover, I could not map the tests with Jenkins by comparing the
release-timestamps, since some releases are 3-4 months apart, while some
are in the span of few weeks.
Releases of the test suite are used only internally by CloudBees. The
Jenkins project always uses the last commit from the master branch when
running the test suite. So just ignore these releases. (Jenkins releases
are on a weekly basis.)
Best regards from Germany, too :-)
Viele Grüße,
Ulli
I would be more than thankful if you could give me some insights about
the
relationship between the two.
Best regards from Germany,
Adi
--
oliver
--
You received this message because you are subscribed to the Google Groups "Jenkins
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-dev/op.x69s8lkrsbfict%40arch.
For more options, visit https://groups.google.com/d/optout.
