Paul Millar wrote:
On Thursday 17 March 2005 11:54, Jakob Eriksson wrote:
Maybe one could script something up so that a commit to tests CVS would not be *possible* without a confirmed test pass on Windows 95, NT, 2000 and XP. That'd be the best thing since sliced bread.
CVS supports doing some server-side validation tests before accepting a commit, so it could be done; but I don't think this would be "nice". CVS is a mechanism for sharing code: its not really a testing framework.
I didn't mean exactly on the CVS level. When Alexandre commits any patch, he first checks that the code still passes regression tests.
I want something similar for the test patches. Maybe like this: the testing dispatcher signs a working patch* with GPG. (Or no GPG. Just set a flag somewhere. Details are not important.)
Alexandre will see this flag when saving a patch from [EMAIL PROTECTED] and know that the patch is OK as far as the test grid is concerned.
It could be as non-intrusive as this: the test dispatcher monitors the [EMAIL PROTECTED] for patches. As soon as it sees a patch it recognises and knows it has tested, it sends a mail to wine-patches akin to:
The patch with CRC32 so-and-so posted by him or her, named so-and-so is hereby verified by me, the Wine Regression Grid Tester. **
or
The patch with CRC32 so-and-so posted by him or her, named so-and-so failed under the following versions of Windows; bla bla blah, with the following error message: blah blah bla some more. Truthfully yours, the Wine Regression Grid Tester. **
Then it's up to Alexandre if he wants to commit a test which the grid tester has rejected, or for which there is no confirmation.
If you don't like the idea of a program spamming wine-patches, it could be separate list, or a webpage with a copy of wine-patches, with different messages colour-codes updated as they get tested by the grid tester.
* A working patch is a patch that has been tested and found working on Win 95, 98, ME, NT4, 2000 and XP.
** Could we call it WineGrind? :-)
For the testing framework, I'd say what we have just now is fine. It lives outside (and on top of) CVS. Having broken tests is OK, provided they're fixed within a suitable time-scale [*].
Actually, I think having broken tests is not OK. It not only goes against my zealotry for Extreme Programming, it's also very annoying when I have _no_clue_ how to fix a broken test and the author is missing or don't want to touch the code with a ten foot stick.
(just my 2c-worth again)
Cheers,
Paul
[*] -- Of course, what is the "suitable time-scale"? who's willing to make sure things get fixed?
Exactly, I feel rage everytime I see those red and yellow boxes at http://test.winehq.org/data/ ;-)
regards, Jakob
