Hello Robert and Jelmer, and thanks for your replies. On Tuesday 22 March 2011, Jelmer Vernooij wrote: > On Tue, 2011-03-22 at 10:48 +1300, Robert Collins wrote: > > On Tue, Mar 22, 2011 at 8:41 AM, Stefano Lattarini > > > to its suboptimal documentation. So I'm going to ask: Robert, as > > > the main proposer/supporter of the SubUnit protocol here, would you > > > be willing and ready to help me out during my prospective work with > > > GSoC, if I update my application's goal to read "Support SubUnit > > > (and also TAP as derivation) in Automake-generated testsuites"? > > > Maybe we could also improve SubUnit's documentation along the way, > > > which would help both me and Automake, and also improve SubUnit > > > itself and make it more "palatable" for potential adopters. > > I'll be delighted to help by clarifying things or making sensible > > small improvements. We should structure things so there is a clear API > > between whatever automake needs and subunit providing it (this allows > > subunit to continue to improve without automake needing to change > > futher, and vice versa). I would offer to make large changes if > > needed, but realistically, work is in the middle of a crucial project > > that is going to be going on for ~ 3-4 more months, so I am somewhat > > time constrained :). That said, Jelmer Vernooij may well - he has > > written a number of Perl subunit tools in the samba project, and they > > are likely to be at least somewhat relevant to whatever automake needs > > are here. > I'd be happy to help with inquiries about subunit where I can, or > co-mentor, whatever works best. > The former would already be great and very helpful -- thanks!
About the latter, that would be great too, but I guess I'm in no position to vouch for it, or to judge if it's really required. Anyway, I see you've already been a GSoC mentor for four years now, so I guess you are able to decide what is necessary or advisable here, and in case, undertake the necessary steps. I'll defer to your judgement in this matter. > FWIW the Perl subunit modules shipped with subunit and included in Samba > are sufficient for parsing existing subunit streams and for generating > subunit streams, but they're still pretty basic. They don't yet > integrate with any of the standard perl testing frameworks like > Test::Simple or Test::More and they (currently) lack support for some of > the newer subunit features. > Thanks for the information. Still, I'll only be able to take clues and/or "inspiration" from them: their direct use is not viable because Automake-generated Makefiles cannot assume the presence of Perl; the best we can do is go with shell + sed + awk ... In the meantime, I've updated my application and roadmap to reflect the new SubUnit focus. Find them attached if you're interested. Thanks again, Stefano
Student Identification ====================== Name: Stefano Lattarini Email address: stefano.lattar...@gmail.com The name of the project ======================= automake - Interfacing with the test protocols TAP and SubUnit Abstract ======== The Test Anything Protocol (TAP) and the SubUnit protocol are simple text-based protocols that allow communication between test scripts and a test harness. The primary target of this project is the implementation of a TAP-consumer and a SubUnit-consumer testsuite harness in Automake-generated Makefiles. This way, a software package using Automake and having a scripts-based testsuite would be able to run multiple checks in each test script, and communicate the result of each of those checks separately to the test harness, which could then issue more faithful statistics about the number of failed/passed/skipped checks. A secondary objective, which will be pursued only if the completion of the primary one will leave enough time, is to enhance Autotest-generated test scripts to allow them to become TAP and/or SubUnit producers. The parallel-tests driver and its limits ======================================== In the recent years, Automake's built-in support for the generation of testsuite harnesses has become quite sophisticated. In particular, the "parallel-tests" driver has performed very good. That driver is targeted at testsuites comprised of test scripts which are mostly independent one from the other, and safe to run in parallel. It is implemented in portable make (obviously with the contribution of portable POSIX utilities, such as the shell, awk, sed, etc.), but still manages to offer many useful and pretty advanced features (please refer to the Automake documentation for more details). Still, the parallel driver still has one major limit. In a scripts-based testsuite, it's common for a single test script to run checks for multiple (usually related) features, and/or against multiple bugs at once -- especially in case these checks require a similar (or even identical) setup. And while it would usually be feasible to separate such related checks into different test scripts, that might require a too-high overhead, either in test execution time or in programmer coding time. The problem with the current Automake parallel-tests driver (and also with its precursor, the "simple-tests" driver) is that the results of all the checks run by an individual test script are hopelessly condensed into the single final exit status of the script. So there is no way to distinguish between, say, a test script that failed because one check out of N in it failed, from one which failed because all the N checks in it failed. Possible enhancements with the TAP and SubUnit protocols ======================================================== The Test Anything Protocol (TAP) and the SubUnit protocol are simple text-based protocols that allow communication between test scripts and a test harness. They allow individual tests (TAP/SubUnit producers) to communicate test results to the testing harness (TAP/SubUnit consumer) in a language-agnostic way. What is important for us is that, using one of these protocols, a single test script can safely run multiple checks and report the outcome of each of them separately to the harness; the results will then be properly displayed and accounted for by the harness, with the right level of granularity. So, for example, if we have a properly written TAP-producing test script that fails because two checks out of 20 in it failed, the test harness won't count that as "1 failure, 0 passes", but as "2 failures, 18 passes". Thus, Automake-generated testsuite harnesses could become TAP and/or SubUnit consumers, for third-party testsuites and for Automake-generated parallel testsuites alike. Also, SubUnit can be seen a "superset" of TAP (while *not* being a TAP extension in a strict sense), and it's possible to convert TAP into SubUnit without too much difficulty (the SubUnit distribution offers a simple python class that is able to do so). Thus, if we manage to make Automake-generated test harnesses able to parse SubUnit, the insertion of a simple filter acting like a TAP->SubUnit converter will almost automatically make them able to understand TAP too. Finally, Autoconf's Autotest currently use its own custom output to report test results. It could benefit of better interoperability if it starts supporting TAP and/or Subunit; in particular, the integration with the Automake-generated harnesses would be greatly improved (and probably also simplified). Some useful links on TAP: * <http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod> Documentation for the TAP format understood by the perl module Test::Harness (kind of "de-facto standard"); * <http://testanything.org/wiki/index.php/Main_Page> A wiki on TAP, with useful discussions, pointers, and also bits of history. * <http://www.eyrie.org/~eagle/software/c-tap-harness/> A C library implementing both a TAP producer and a TAP consumer; should be quite mature and actively maintained. The SubUnit homepage: * <https://launchpad.net/subunit> contains some basic documentation on SubUnit, and SubUnit producers for various languages (python, perl, C and C++). The Samba project is an early adopter of SubUnit, using it in (at least part of) its testsuite: * <http://jelmer.vernstok.nl/blog/archives/262-subunit-usage-in-Samba.html> * <http://samba.org/~jelmer/samba4-testing.pdf> Benefits of this proposal ========================= We have already clearly described how the use of TAP/SubUnit would improve the granularity in the presentation/summary of test results. The TAP/SubUnit model also offers what I deem to be a better mental model of the typical "testing with test scripts" scenario: a testsuite contains test scripts, each of which can run multiple checks. The smallest meaningful piece of result is the outcome of a check, and *not* the exit status of a test script. This model is both more granular and more natural. TAP is well-established in the Perl community, and thus the protocol have already proven itself as valid "in the real world". This seems confirmed by the existence of quite many third-party TAP producers and consumers for different languages. SubUnit is admittedly not yet well-established as that; still, being designed with TAP compatibility in mind, it should share many TAP qualities. One advantage it has over TAP is that it's meant to work well with more modern programming languages (e.g., python) and more widespread testing paradigms (xUnit). Last but not least, it is already used by at least an important real-word project like Samba. There are already C/C++ libraries aimed at writing TAP-producing and/or SubUnit-producing test programs. Thus the addition of the new TAP and/or Subunit support in Automake would allow an easier and more proficient integration of custom C unit tests into Automake-generated testsuite harnesses. This would be particularly useful, considering that projects based on C and/or C++ are the primary client base of Automake. Finally, the TAP/SubUnit support in Automake could rely on the existing implementation and APIs of the parallel-tests driver, extending them in a mostly (if not even complete) backward-compatible way. The new functionality would thus build on a well-established foundation, and place itself in a thriving paradigm. Deliverables ============ Once this proposal is implemented, Automake should be able to generate TAP-consumer and SubUnit-consumer testsuite harnesses that work portably on many systems (ideally all those Automake strives to support) and with many vendor make implementations (ideally all those Automake strives to support). Failing this is due time, the "fallback plan" is to have the generated testsuite harnesses either: - support only one among SubUnit and TAP; or - support both, but with the additional requirement of perl (>= 5.6 should be enough) being installed on the target system; Finally, if there is enough time left, Autotest might be enhanced so that the test scripts it generates become able to produce TAP and/or SubUnit output. This is not a primary target, though, and might as well dropped if it's clear there won't be enough time to complete it. Plan and Roadmap ================ Q: How will you and your mentor track your progress as you work on the project? A: Since technical discussions about design and implementation of new Automake features take place on the public lists <automake@gnu.org> and/or <automake-patc...@gnu.org>, most measurable progresses are publicly available there, and thus quite easy to judge. Also, every new feature going in Automake must have proper testsuite coverage, so that we can be confident that the new code will really works. Finally, the project mentor has been the Automake maintainer since 4/5 years now, so that he's surely able to judge how a partial implementation if faring, and what the chances are for it to be completed in time. Q: How the mid-term evaluation of your project will be made? A: I must have an implementation of Automake-generated TAP-consumer testsuite harness that works at least with non-obsolete versions of GNU make at least on GNU/Linux, Solaris 10 and FreeBSD 8. This implementation must comprise test cases (maybe non exhaustive) and documentation (also unpolished, but basically complete). Q: Remember to mention any periods during the summer when you won't actually be available to work on the project (though remember, the Summer of Code project is expected to be your main activity). A: I plan to be available for basically all the period of the project. If not the whole day, at least half a day, basically each day (either morning and early afternoon, or later afternoon and evening). There might be *very few* short periods (one, max two days) when I'll be completely off-line and "away from keyboard" -- I'll let the mentor know about them in advance, obviously. Communication ============= First, I'm already in contact with the mentor, due to my on-going involvement with the Automake project. I've also been in contact with the backup mentor, due to my (small) contributions to Autoconf. In the Automake and Autoconf projects, discussions about design and documentation, and proposals and reviews of patches are public, taking place on the projects' pre-existing and well-established mailing lists. The mentor reads them regularly, so that posting patches an ideas there is a sure way to have them considered and reviewed by him (and possibly also by other mailing list readers, which might contribute with their own useful insights, questions, and experiences). Finally, my experiences have showed that real-time interaction with the mentor to work on a particular issue (either on-list or off-list) is not only possible, but also efficient and effective. Qualification ============= I'm quite familiar with the policies of both the Automake and Autoconf projects (especially coding standards, patch review process, and portability concerns). Moreover, the copyright assignment to the Free Software Foundation that allows me to contribute code to the Automake and Autoconf official repositories is already in place (and has been for more than a year and a half now). I've started contributing to the Automake project an year and a half ago. During the last year, my involvement with the project has steadily increased: I've posted more patches and participated more actively to discussions on the mailing lists, I've been granted commit rights, and I've started helping with the bug tracker and related issues. I plan to continue my involvement with Automake in the foreseeable future. I have access to some systems to test my work on: Debian GNU/Linux (with custom installation of older versions of tools and compilers), FreeBSD 8.0, and Solaris 10. This can help me in finding out early potential portability problems. I believe that powerful, easy-to-use testing frameworks are vital for the development of modern software. Also, I find software testing an interesting, and sometimes even fascinating, topic. So I can say I'm motivated enough to work in this area.
Tentative roadmap for GSoC project "automake - Interfacing with a test protocol like TAP or subunit". 1. COLLECT INFORMATION: Before April 20 ---------------------------------------- Read carefully the TAP and SubUnit specifications available. Since the SubUnit specification is somewhat lacking, in case of doubts/problems ask to the authors/supporters (they've already proven themselves willing to help in this regard). Take a look at real-word usages of TAP (mostly the distribution of Perl and some major perl packages) and SubUnit (mostly the Samba testsuite); familiarize as much as possible with the corner cases and the "rough edges", jot down some notes about things that are unclear might be deemed to cause potential problem. Finally, if there's still time, take a look at existing implementations; chiefly: - for TAP, Test::Harness and c-tap-harness; and - for SubUnit, the subunit package and some support code in the Samba testsuite. All the licenses involved are GPL-compatible, so this should be OK from a legal point of view. 2. EXPERIMENTING: April 20 - May 15 (Before the official coding time) ----------------------------------------------------------------------- Write scripts (better if in pure awk, for performance reasons; but shell scripts are acceptable) and Makefiles that implement TAP and/or SubUnit consumers (it's OK to do so by tweaking Automake-generated Makefiles, and it's ok if they're not 100% functional in this phase). Try to make those examples as portable as possible, to spot potential portability issues early. Test at least on GNU/Linux, FreeBSD 8 and Solaris 10, and their native make, shell and awk implementation(s). Exchange early ideas and doubts with the mentor, and, if appropriate, also with the official mailing lists. If there's still time, take a deeper look at Autotest documentation, implementation, and history, as to be able to later (try to) extend it to produce TAP and/or SubUnit compatible output (time permitting). This phase should help with the subsequent "real" implementation, by providing some ideas, and quite likely also code (incomplete of course, but probably good as a starting point and/or for "cut & paste"), and tentative testcases. 3. BASIC DESIGN: May 15 - June 5 (Official coding period starts May 23) ------------------------------------------------------------------------- We'll have to decide which is the best way to integrate the TAP/SubUnit consumer code into Automake -- will we use new primary, a new option, or maybe some new special syntax and/or variables? This is something to be discussed publicly on the Automake mailing lists. Also, how much of the parallel-tests implementation and scaffolding can be reused? What about parallelization and/or synchronization issues? What about other issues that came up in the experimentation phase? All these doubts should be worked out. If there's time, we might also start planning how Autotest can be extended to allow it to produce TAP/SubUnit output. This is something to be discussed publicly on the Autoconf mailing lists. BTW, at this point, we'll probably need to go back and do some more experimenting (but more purpose-oriented than in the previous phase). And we could also write some early test cases and documentation. 4. WRITE BASIC TESTCASES AND DOCUMENTATION: June 5 - June 12 ------------------------------------------------------------- Documentation about design and general concepts shouldn't be long nor verbose, but it's important to get it right and to make it as clear as possible -- which can be tricky, and take some time. Also, condensing the most important use cases into tests is not immediate (although it shouldn't be very difficult in this phase). 5. CODING, PART 1: June 12 - July 6 (Until mid-term evaluation) ---------------------------------------------------------------- Here I'll have to extend Automake to provide TAP and SubUnit aware test harnesses. The resulting code must behave as to match the existing documentation and design. It should also be able to generate Makefiles that work on at least GNU/Linux, FreeBSD and Solaris, with their native make, shell and awk implementation(s). In case I fail to accomplish this in due time, here are my fallback plans: - support only one among TAP and SubUnit -- but support it portably! or: - support both TAP and SubUnit, not fully portably -- e.g., relying on a Perl interpreter to be installed on the target system. This is not a clear-cut phase. During it, I'll have to be ready to extend and adjust the new documentation; adapt the existing testcases and add some more (probably many more); read and understand parts of the Automake codebase I'm not familiar with yet; and maybe doing more experimentation. In the very worst case, the mentor and I might even be forced to revisit and modify our previous design. 5. CODING, PART 2: July 6 - July 23 ------------------------------------ If the implementation for TAP and SubUnit aware test harnesses written in the previous phase is complete and wholly functional (which include portability to at least BSD and Solaris make, and independence from Perl, but might allow for some rough edges), in this phase I'll try to enhance Autotest to enable TAP and/or SubUnit compatible output in its generated test scripts. Otherwise, I'll have to work more on Automake, ditching the plans for Autotest enhancements. 6. DOCUMENTATION POLISHING: July 23 - July 28 ---------------------------------------------- Polish/complete documentation of all the implemented features. Maybe add some examples, and if there's time contribute them to existing third-party tutorials, FAQs and/or wikis about the autotools. 7. STRESS AND PORTABILITY TESTING: July 28 - Until done -------------------------------------------------------- Test the code on other systems which Automake strives to support (e.g., Cygwin, MSYS/MinGW, AIX, HP-UX, IRIX), and (try to) fix all the problems and incompatibilities that show up. This might require some more help from the mentor, which has access to an impressive array of testing systems (all the major proprietary Unix implementations are among them). 8. FINAL CLEANUP: All the time that's left (if any) ---------------------------------------------------- Note that this last pass is not strictly necessary, and might be omitted altogether in case we're out of time. Here we could add more tests about corner cases; polish documentation and/or code; and maybe even convert (part of) Automake's own testsuite to the use of the newly implemented TAP/SubUnit-based harness.