On Thu, Sep 28, 2017 at 2:51 PM, Stephen Connolly <[email protected]> wrote: > writing good acceptance tests is a high skill and not something that > can be easily crowdsourced.
Agreed. In particular we should not be asking GSoC contributors to write new acceptance tests. Every PR proposing a new test should be carefully reviewed with an eye to whether it is testing something that · could not reasonably be covered by lower-level, faster tests · is a reasonably important user scenario, for which an accidental breakage would be considered a noteworthy regression worth holding up or even blacklisting a release (of core or a plugin) · is sufficiently distinct from other scenarios already being tested that we think regressions might otherwise slip by > there is a fundamental flaw in > using the same harness to drive as to verify *because* any change to that > harness has the potential to invalidate any tests using the modified > harness Probably true but does not seem to me like the highest priority facing us. > it is all too easy to turn a good test into a test giving a false > positive I am not personally aware of any such historical cases (maybe I missed some). The immediate problems are slowness, flakiness (tests sometimes fail for no clear reason), and fragility (tests fail due to trivial code changes especially in the UI). > We need an ATH that can be realistically run by humans in an hour or two Yes that seems like a good goal. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr04H5Zd4xoq2HHQH0XdUezCOnykcOgRM_9T7npwtwBA0Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
