RFC: CPAN::AutoTestReporter
SYNOPSIS
I propose a module that will automatically allow a distribution author and
the CPAN Testers group to receive a configuration summary and testing
report (ala cpantest) of the testing process of a distribution build.
Participation in this process would be strictly optional.
Note: I've searched through the [EMAIL PROTECTED] archive, the
[EMAIL PROTECTED] archive as well as comp.lang.perl.modules and found
virtually no relevant prior discussion on this topic.
TESTERS.CPAN.ORG
testers.cpan.org states "CPAN Testers can always use more people willing
to spend the time and effort testing new modules as CPAN keeps growing
daily."
What better way to accomplish this than by involving the community
at large by allowing them participation in the process in a behind-the-
scenes, automated manner?
testers.cpan.org also states "The ultimate goal is to improve the
portability of the distributions on CPAN, and provide good feedback to
the authors."
We have thousands of thousands of people already testing these
distributions on a frequent basis and we don't get their test results! Is
it just me or is something wrong here? Shouldn't we be taking advantage
of all this "unreported" testing to achieve our goal?
DESCRIPTION
The module would do things approximating the following:
* Allow an end-user (i.e. someone building a distribution) to have their
perl configuration summary (perl -V) and test report (output of make
test) automatically collected and submitted. The report would be sent
via e-mail to [EMAIL PROTECTED] and the distribution author and
would contain the same information that cpantest does
* Their testing process would be monitored and processed as follows:
- If Test::Harness/Test reports all tests passed it will generate the
same e-mail as cpantest does under the same condition
- If Test::Harness/Test reports that not all tests passed it will generate
the same e-mail as cpantest does under the same condition and place
the output of 'make test' in the comments field
- If there are no tests defined, it will generate the same e-mail as
cpantest does under the same condition, -or- we could leave this
case unhandled and "force" distribution authors to include tests if
they want statistics and perl configuration summary data in return
- 'na' would not be handled, and is probably best left to manual
testing via CPAN Testers
INTERFACE
The user building the module could get something like:
"For the purpose of improving the quality and support of this distribution,
the CPAN Testing service and the author of this distribution are collecting
the test data and configuration details of this distributions testing
process. This information will be sent via e-mail to the CPAN Testing
service and the distribution author. The only information that will be sent
is the output of 'make test' and your perl configuration summary (perl -V).
This does not interfere with the distribution in any way, nor receives
any information back. We are interested in this information for statistical
purposes and to improve support for various operating systems. You are not
required to contribute to this effort."
Report your configuration details and make test output (yes/no) ?"
RATIONALE
There are several reason why I think this would be rather useful:
* It benefits the distribution author, by letting the author know:
- How widely used their distribution is by the general public. They could
use this information to prioritize how they spend their development time
- Which versions of their distribution are getting used by the general
public
- Which platforms their distribution is being used on by the general
public. They could use this information to add better support and tests
for various platforms
- What versions of perl their distribution is running on by the general
public. They could use this information to better support old perls.
* It benefits the CPAN Testing service for the following reasons:
- Gives us a veritable army of potential new testers, and we could
certainly use the assistance
- This module fits perfectly into the already established framework
- Supplements the manual work that we CPAN Tester's do
- Lends more automation to the testing process, which is something we
are striving to accomplish
* Distribution authors have been asking for something like this for quite
some time. They get automatic feedback, and some statistics on usage and
we get tests results. Everyone wins.
TESTING CATEGORIES
We are currently bringing many resources to bear on the testing and quality
issues. We currently have tools that fit into three categories. Manual
testing (eg: cpantest), semi automated testing (eg: CPAN::Report), and
fully automated testing (the smoke modules). These categories are all
extremely important yet operate with very few actual testers, and could be
well supplemented by the semi automated testing of hundreds, or thousands
of users
CAVEATS
Obviouslly, this module would require that the machine that the module
is being built on has a network connection and could send mail.
FAQ
"If a distribution author wants some usage statistics, just have them put
'If you use this distribution, send an email to [EMAIL PROTECTED]' in the
POD of their distribution"
Infact, this is precisely what happens. The only difference is that it's
automated. When was the last time you got an e-mail from someone just to
say they were using your module? The benefit is that a person doesn't
have to spend the (albeit miniscule) amount of time to send an e-mail
and that the e-mail will contain useful information.
"How does this differ from what the CPAN Testers already do?"
It expands our pool of testers. It supplements what we do manually.
It strives to achieve more automation in the testing process. It
allows the community at large to contribute to the quality of CPAN.
"If people want to participate in the testing process, they should just go
to testers.cpan.org, subscribe to the mailing list, download the script
and start sending in test results"
Many people don't have the time or inclination to become a CPAN tester
and test as many modules as they possibly can on as many platforms as they
possibly can. Many people don't want the hordes of mail from the list.
However, most people who build modules seem to be willing to submit the
test results for the modules that they build. The beauty of this process
is that it's all done automatically, behind the scenes, and all the
involvement they have is typing "yes" at the prompt.
THE NAMESPACE
If someone has a better suggestion than CPAN::AutoTestReporter, I'm
all ears.
CONCLUSION
Thoughts? Comments? Suggestions? Feedback? Thanks for reading!
--
Adam J. Foxson
CPAN Tester for Darwin/MacOS X
