Re: Add a "random" target to bsd.regress.mk

2017-03-22 Thread Scott Cheloha
> On Mar 22, 2017, at 11:47 AM, Alexander Bluhm wrote: > > I would not do all this work. I have indepentent targets for some > of my own tests that I use regualry for development. But for other > people's tests which just pass, I don't care. Making the individual >

Re: Add a "random" target to bsd.regress.mk

2017-03-22 Thread Alexander Bluhm
On Tue, Mar 21, 2017 at 08:56:02PM -0500, Scott Cheloha wrote: > > Do you plan to improve this situation? > > It depends. Do you think improving the independence of each of those > cases is valuable in and of itself? I would not do all this work. I have indepentent targets for some of my own

Re: Add a "random" target to bsd.regress.mk

2017-03-21 Thread Scott Cheloha
> On Mar 20, 2017, at 9:13 AM, Alexander Bluhm wrote: > > I would expect to find more issues in the tests than in the software. > > Unfortunately I fear a lot of our tests do that. > > Independent tests cases are useful when fixing tests or doing test > driven

Re: Add a "random" target to bsd.regress.mk

2017-03-20 Thread Scott Cheloha
> On Mar 18, 2017, at 8:48 PM, trondd wrote: > > [...] > > Actually, rereading what I quoted, I see you're concerned with unmasking > false positives, which can be manually rerun in isolation to reproduce. > My thought still stands for the flip-side where a test fails

Re: Add a "random" target to bsd.regress.mk

2017-03-20 Thread Alexander Bluhm
On Sat, Mar 18, 2017 at 05:42:20PM -0500, Scott Cheloha wrote: > My thinking is that statically ordered regression runs can mask > bugs in the software under test and the tests themselves. I would expect to find more issues in the tests than in the software. > In general, a test can put your

Re: Add a "random" target to bsd.regress.mk

2017-03-18 Thread trondd
On Sat, March 18, 2017 6:42 pm, Scott Cheloha wrote: > In general, a test can put your system into a state that allows a > subsequent test to pass when it would have otherwise failed. > > > Any takers? Thoughts? > If such a bug is revealed, how does someone rerun the tests in the same order

Add a "random" target to bsd.regress.mk

2017-03-18 Thread Scott Cheloha
Hi tech@, The patch below adds a target, "random", to bsd.regress.mk, as well as a description of the new target to bsd.regress.mk(5). The "random" target is the same as "regress", except that the ordering of the regression targets is randomized each time. My thinking is that statically ordered