On 17 Sep 2012, at 17:25, David Boyes wrote: >> 'make check' on a single machine will never give you useful testing results >> other than to find packaging or "smoke test" errors, which aren't all that >> helpful overall.
I agree with you with regards to crowd sourced testing, but I just wanted to call out this statement, as I think make check can actually be hugely helpful. In the past, we used to have the problem that changes would get committed to master that would cause the build to fail. So, if you were working on new stuff, and had just pulled in a recent upstream to work on, you often ended up having to spend lots of time just getting the tree to work again on your chosen platform. Buildbot has pretty much eradicated this problem - I can be sure that git fetch will give me a tree that works. So now, the new problem is that changes in master may well be unstable. Instead of working on building your shiny new feature, developers end up having to track down why a particular binary segmentation faults and so on. Applying 'make check' as a commit requirement, and improving its coverage will hugely help with this problem. I see it more as a developer tool than a general QA solution. What its not going to do is replace large scale test harnesses and detailed test plans. We still need the ability to generate large numbers of clients hammering a server to expose particular problems. Over the 1.6 cycle, the majority of this work was done for OpenAFS by volunteers at European HPC sites, who used down time on their clusters to generate particular types of load against our fileservers. This led to a number of long standing bugs in both the demand attach fileserver, and the RX stack itself, being identified and fixed. Cheers, Simon. _______________________________________________ OpenAFS-info mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-info
