On Mon, May 18, 2015, at 02:27, Kamil Rytarowski wrote: > What do you think of introduction in the build machines randomized > configuration of mk.conf(5) in each build?
While it'd certainly be nice if the non-standard options I use were more regularly tested, I think the overall value of the automated builds and tests is much higher if they are reproducible. Randomizing the test conditions might turn up some bugs that otherwise go unnoticed, but it would make it impossible to narrow down which set of changes caused a particular failure, because we could never be sure whether the difference between a successful build and a failed one was in the source code or in the build options. That said, I think we may be confusing two different things here, the build clusters that provide up-to-date compiled binary distributions, with the standard options, that people can just drop onto a drive and try out; and the test machines that build and run tests in search of regressions. I don't think that randomized configurations make sense for either of those, but the arguments are a bit different in the two cases. Now if we had enough resources to test non-standard configs *systematically*, that might be interesting, but in my experience breakage related to my non-standard config is so much rarer than breakage/fixage in the standard builds that I doubt it'd be worth the effort. -- IDL
