Dear Thomas and Christoph,
> As far as we
> know GUI Tests with STF are only run nightly on the master branch. All
> other JUnit Tests are run on any patch set submitted to code review.
> This might result in Gerrit patch sets passing regular JUnit tests and
> therefore get marked as verified by Jenkins CI although they break STF
> test cases.
Even worse: The JUnit tests are executed for every new patch version (Job
"Saros-Gerrit"). The eventual submission of a patch is not tested immediately.
Even though a patch may be mergeable without a syntactical conflict, it may
produce a defect on a different level thus breaking both the STF and the JUnit
tests.
But thankfully the JUnit tests are executed every hour on the master branch
(Job "Saros"). And there is also the nightly STF job ("Test_Saros_STF").
> Goal: Analyze schedule and results of automated test runs
I like your Goal, but consider the following background information:
During the last semester break we conducted a project ("SWTP" -
Softwaretechnik-Projekt). The participants of this project worked with a clone
of the normal Saros repository. We played around with a different approach
using feature branches (the idea came from a blog post [1]):
* The master branch is closed to direct commits; everyone works on feature
branch instead.
* New patch versions are verified by Jenkins in the usual way (JUnit tests)
* As soon as a patch is finally submitted to a feature branch, a Jenkins
integration job will try to merge the feature branch (containing this new
commit) back into the master branch. This is done by the following steps:
** Merge on a textual level (normal "git merge"). If it fails due to merge
conflicts, the integration fails, and the master and the feature branch will
diverge. Otherwise:
** The source code is compiled. If it fails, the integration fails. Otherwise:
** The JUnit tests are executed. If they fail, the integration fails. Otherwise:
** The re-integration succeeded. The resulting merge commit is pushed back to
the official repository.
* The deviation of master and feature branches poses no immediate problem:
every subsequent commit on the feature branch may resolve the issue.
* Main benefit: developers working on different topics do not heavily block
each other. It's up to the developer when he/she wants to "fix" the "broken
merge": It could be done immediately to prevent further deviation of the
branches, or the developer decides to add more commits to the feature branch
before reintegrating them into the master.
The (currently inactive) integration job is named "Saros-SWTP-Integration". In
case you wonder why this approach has not been adapted in the actual Saros
workflow? It's pretty simple: the current version of Jenkins' Git plugin does
not provide the "--no-ff" option for merges. Therefore fast-forward merges may
occur, which make the Git history really hard to read, because there are no
easy-to-spot swimlanes.
For the near future I see to possibilities to cope with that: (1) Provide a
patch for Jenkins' Git plugin to fix this problem once and for all; (2)
Circumvent the pre-build merge feature of Jenkins' Git plugin by
re-implementing its functionality in a shell script, which would be embedded in
the Jenkins job. The required human resources for either of these solution are
yet lacking, but not for long :)
You may want to reconsider your Questions with this background information in
mind?
For rebuilding the Jenkins setup on your own machine I'm not the best contact
person.
@Stefan: Could you take over this topic?
Regarding your second Goal "Improve code quality" please not that code quality
is not a "mono-lithic" property, it's rather complex or facetted. You may want
to look into [2] (reachable via our university's proxy), where the following
eight quality facets are listed:
* Functionality/Capability
* Performance
* Reliability
* Usability
* Maintainability
* Transparency
* Portability
* Testability
Unfortunately these facets are correlated, e.g. performance is hard to improve
while keeping the other ones at the same "level".
Furthermore your Questions a rather simple to answer:
> - Is it possible to improve code quality and reduce code complexity?
> - Is it possible to improve the maintainability of the source code?
Possible? Yes :)! So: What do you *really* want to know?
Best Regards,
Franz
[1]
http://twasink.net/2011/09/20/git-feature-branches-and-jenkins-or-how-i-learned-to-stop-worrying-about-broken-builds/
[2] http://link.springer.com/book/10.1007/978-3-540-76323-9
-----Original Message-----
From: Christoph Viebig [mailto:li...@christoph-viebig.de]
Sent: Wednesday, December 19, 2012 5:33 PM
To: dpp-devel@lists.sourceforge.net
Subject: [DPP-Devel] GQM project
Dear Saros Community,
in our university course about software processes at Freie Universität
Berlin we are now working on Goal Question Metrics (GQM).
As part of our exercise course we have to elaborate a study on GQM in an
open source project which we would like to conduct at the Saros
project. Therefore we have written down some ideas.
GQM is a software metric model and therefore aims to measure metrics of
software processes and products to allow comparison and evaluation of
such. As a result either changes can be suggested or current procedure
can be confirmed.
Our first idea regards the continuous integration service. As far as we
know GUI Tests with STF are only run nightly on the master branch. All
other JUnit Tests are run on any patch set submitted to code review.
This might result in Gerrit patch sets passing regular JUnit tests and
therefore get marked as verified by Jenkins CI although they break STF
test cases. Hence the first goal is to analyze the schedule and results
of automated test runs.
Goal: Analyze schedule and results of automated test runs
Object of study: Unit- and STF-Test-Results of all (accepted) patch sets
Focus: Number of broken test cases in all (accepted) patch sets
Stakeholder: Saros developers
Questions to answer for this goal:
- Is the current schedule of automated test runs adequate?
- For which patch sets or branches and at which time are test runs
useful?
Our second goal targets Saros code quality and is a very general one.
Goal: Improve code quality
Object of study: Saros source code
Focus: Code complexity
Stakeholder: Saros developers
Questions to answer for this goal:
- Is it possible to improve code quality and reduce code complexity?
- Is it possible to improve the maintainability of the source code?
As this is still a draft we would like to ask if you have comments on
the proposed goals and questions. We are not familiar with Saros for a
long time yet so it might happen that some of our goals or questions are
not very useful to be asked. Please tell us! Are there any which would
be more important to the project?
To further investigate possible metrics to answer our questions
regarding CI test workloads we have setup an instance of Jenkins. We are
now at the point where we have to configure Jenkins to run JUnit and
STF-Tests on specific branches of our Saros repository.
What actions are needed to configure this? Can you give us the
configuration you use? Would it be possible to run STF-Tests without
Jenkins, i.e. from the Shell as well?
Thank you very much in advance!
Best regards
Thomas Benndorf and Christoph Viebig
------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
DPP-Devel mailing list
DPP-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dpp-devel
------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
DPP-Devel mailing list
DPP-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dpp-devel
