This has been a very useful thread and brings up great debate regarding the "functional specification" part. I agree that the test harness can be very powerful in make verification - even across language bindings with some of the idiomatic work.
However what's been dropped from this thread is Rajith's important and valid concerns for users and developers: 2. Helps developers to navigate and understand the code more easily. 3. Finally we need this documentation sooner than later to accompany our releases to for the end users I think the test harness is marginally useful in addressing these concerns. William ----- Original Message ----- > On Tue, Sep 25, 2012 at 4:25 PM, Rob Godfrey > <[email protected]>wrote: > > > > The reason the python tests *won't* work as a spec are precisely > > > because they cannot "correspond to a piece of functionality". > > > > > > > > I'm not sure that you have put forward a case that they "can not" > > only that > > they "do not". I agree that the current tests are a fair way from > > meeting > > the requirements of a functional spec, but fundamentally there > > should be a > > 1:1 correspondence between a functional spec and a test suite. > > Whether one > > defines the spec explicitly (and then writes an implicit test > > suite) or > > implicitly (via the mechanism of a well defined set of tests) is > > really a > > style issue I think. > > > > > > > The tests are really good at what they do: verifying that > > > functions > > > work as expected. Keep them coming. > > > > > > > > Actually I disagree. The tests currently aren't all that great. > > When trying > > to get the tests to pass against the Java codebase there's really > > no help > > in trying to work out what the tests are supposed to do (though > > some - such > > as against the message API are much better by having a much clearer > > definition of what it is that they are testing). > > > > > > > But the tests are lousy at specifying what can be done. I pulled > > > up > > > messenger.py as a random example. > > > 1. Not a single comment. > > > > > > > +1 > > > > > > > 2. Magic numbers. > > > > > > > +1 > > > > > > > 3. Magic process flow (setup, teardown part of some framework > > > unrelated to messenger. What's with assert? What's the thread > > > model?). > > > > > > > +1 > > > > > > > 4. No enumeration of functions to call, function arg lists, arg > > > defaults, arg units, and so on. > > > > > > > Not sure exactly what you mean here, though if you are saying that > > the > > tests should in some way be categorized by which API function they > > are > > testing, then I completely agree. > > > > > > > 5. What are prerequisites? > > > > > > As a counterexample to using tests as specs, take a look at > > > cpp/src/tests/acl.py and tell me how C++ broker ACLs work. > > > No amount of ACL python test code will be a decent spec from > > > which users can understand and write ACL rules. > > > > > > > > I'm not sure that one can prove a negative by counter-example. My > > personal > > view is that we *can* define a spec implicitly through tests, but > > that we > > haven't yet proved that conjecture. Ultimately a (functional) test > > suite > > and a functional spec should have some sort of one-to-one > > correspondence > > and if they are defined in separate forms then there is a risk of > > divergence (and code is generally more precise than text). However > > for many > > people (probably including myself) it is easier to first write the > > positive > > statement (the spec) and then define the negative (via the test). > > > > Ultimately the means (test vs. doc) is less important than just > > ensuring > > that there is an adequate definition somewhere, which - to Rajith's > > original point - I'm not sure there is yet (at least not written > > down). > > > > I think the term 'functional spec' is possibly a bit distracting > here as > different people have different ideas of what that means, so let's > forget > about that for a minute and look at what we actually have and how it > is > working/not working. > > 1. We have an interface definition (in python), and a test suite/TCK > (in > python). > 2. We have two implementations of this interface, the C > implementation, and > the Java implementation. > > On the one hand this is a very typical arrangement: An API, a test > suite, > and a set of implementations. On the other hand this is somewhat > unusual in > that the interface definition and test suite are in a different > language > than either implementation. > > What's incredibly powerful about this is that we have the ability to > enforce with code consistent behaviour between the two > implementations. > Please don't underestimate how useful this is. We have many years of > experience transcoding protocol implementations from one language to > another without this capability, and I would say we have proven > conclusively at this point that to attempt this kind of thing without > a > common test suite is guaranteed disaster. > > That said, there are some less than ideal things about the current > setup: > > 1. The interface definition is scattered and difficult to see. It's > pretty > much a function of the C header files translated to python via swig, > and > this is not exactly idiomatic for python or Java. > 2. Understanding the interface definition and test suite requires too > much > familiarity with the C implementation. > > I've already taken some steps to address both (1) and (2). Namely > defining > the idiomatic python API, thereby bringing together these scattered > definitions into one place, and started writing new tests in terms of > it > rather than directly against the ad-hoc interface as defined by swig. > > Of course at this point much of the test suite is still written > against the > ad-hoc swig bindings, and the idiomatic python API itself has no > API-doc as > much of that documentation is still living inside the C header files > via > doxygen, and I really don't want to have the same documentation in > two > places. I believe if we move that documentation into the python > interface > definition, however, it would be possibly to extract via pydoc and > related > tools to serve as a source for API-doc for the C and Java impls, and > any > other bindings as well, and right now I think we would be better > served by > having that documentation live in python rather than C (modulo of > course > the C related portions). > > There is of course a back log of work here that can't be done > instantaneously, however I suggest we create a tests component in > JIRA, and > if people run into issues understanding a specific test or a given > aspect > of the interface definition we can file JIRAs there and give priority > to > rewriting those tests against the idiomatic interface as well as > bringing > together all available documentation into the idiomatic API > definition. > > --Rafael >
