Ian D. Leroux wrote: > On Mon, May 18, 2015, at 02:27, Kamil Rytarowski wrote: > > What do you think of introduction in the build machines randomized > > configuration of mk.conf(5) in each build? > > While it'd certainly be nice if the non-standard options I use were more > regularly tested, I think the overall value of the automated builds and > tests is much higher if they are reproducible. Randomizing the test > conditions might turn up some bugs that otherwise go unnoticed, but it > would make it impossible to narrow down which set of changes caused a > particular failure, because we could never be sure whether the > difference > between a successful build and a failed one was in the source code or in > the build options.
Both types of bugs are fatal, aren't they? Like switching on catman and disabling info results in failure (and it's actually true). I see no technical problem in caching used mk.conf. This way we would penetrate all possible configs with just a matter of time including the default one. > > Now if we had enough resources to test non-standard configs > *systematically*, that might be interesting, but in my experience > breakage related to my non-standard config is so much rarer than > breakage/fixage in the standard builds that I doubt it'd be worth > the effort. > OK, anyway I will try to do it occasionally and locally. The catches problems are rather cosmetic, but still halt the proper build.
