On Fri, 2 Nov 2001, Chris Hostetter wrote: > true... i'm not sure what kind of backend there is for storing test data > (after cpantest sends the email, how does the data get into the > testers.cpan.org site ?) but perhaps if the primary DB for test results > was modified to allow multiple results for each > module/perl/osname/osvers/arch pair, it could keep track of things > like "on this type of system, the tests succeeded for 2/2 active testers, > and 230/245 passive testers"
The test results from the emailed reports gets extracted and then stored in a database, which does allow such multiple results. By forming an appropriate query on the database (see the search page on the site), one could extract the success rate for a given distribution according to the system tested. What seems difficult to do in general is to automatically decide whether or not a given test result is useful or redundant. Suppose distribution A has a standard test suite, plus some additional tests depending on if an optional distribution B is available. One person might test A without having B, where it passes. If another person tests A, but has B, and finds the tests fail, such a result would be useful to report, even if it's on the same system as the first report. But there's no way to tell that it's useful except if the first report contained some comments giving details of the configuration used to do the tests, in which case the decision to submit or not would have to be made by a human. > : ftp://theoryx5.uwinnipeg.ca/pub/other/CPAN-Report-0.91.tar.gz > : This allows one, within the shell, to submit reports via a > : "cpantest" method, using much the same syntax as the cpantest > > i haven't look at it yet, but it sounds like a good idea. Does the CPAN > shell save the test results when you (try to) do an install or test ? .. > would this addition allow you to submit the existing test results, or > would it try to turn the tests again? (some of those test can take a > while) The tests won't be rerun - the module saves the dialogue from the last make, test, or install session done thru the shell to a file, which you can then include in the comments section of the report by an appropriate switch to the "cpantest" command. best regards, randy kobes
