Dear all, let my add my perspective on testing an general as I perceive it; maybe this helps to clarify the discussion.
I personally distinguish the following use cases, with decreasing
distance to production:
a) Iterative evaluation of changes to the system
You change something, and run build -a iteratively to figure out what
the real impact is, what has to be amended etc.
Currently, there is no central support for this (cf. one of my previous
e-mails), but that is fine for the moment. Personally, I tend to use a
sophisticated combination of existing TUM machines combines with my
personal machine, that works out quite well in practice.
b) Checking of optimistic changes to the system
You change something under the well-founded assumption that there
shouldn't be any breaking impact. The CI testboard may support or
refute that assumption automatically.
Particularly, if a change has passed the testboard, it has also passed
the quality gate for the main repository. That quality gate ensures
that your change, whatever yet undiscovered effects it might still
produce, does not hinder other people pulling from the main repository
to do their work due to fundamental breakdowns in sessions.
c) Sustainable system governance
After a push to the main repository, there might still be undiscovered
issues, e.g. document production, platform-specific drop-outs, worse
resource usage etc. Hence the regular regression test of the main
repository with wide platform coverage, systematic collection of
statistics etc.
It is important to distinguish b) and c) properly. b) demands agility,
whereas c) demands predictability.
In my perception the current Jenkins infrastructure has a slight bias
towards b); but I guess the framework is flexible enough to cover c)
also, although I am not that involved to tell on the spot what would be
missing here.
Cheers,
Florian
--
PGP available:
http://isabelle.in.tum.de/~haftmann/pgp/florian_haftmann_at_informatik_tu_muenchen_de
signature.asc
Description: OpenPGP digital signature
_______________________________________________ isabelle-dev mailing list isabelle-...@in.tum.de https://mailmanbroy.informatik.tu-muenchen.de/mailman/listinfo/isabelle-dev
