On Wed, Jan 23, 2013 at 6:10 PM, Rob Godfrey <rob.j.godf...@gmail.com>wrote:

> On 23 January 2013 19:09, Rafael Schloming <r...@alum.mit.edu> wrote:
> > I've added another wiki page that documents the proton release steps as
> > best I can remember. I'll updated it more during the 0.4 release:
> > https://cwiki.apache.org/confluence/display/qpid/Proton+Release+Steps
> >
> > I think it's important to understand the overall release and testing
> > process as it is a significant and perhaps underrepresented factor
> against
> > which to measure any proposals. I believe the build system requirements
> > documented below are inherently incomplete as they don't recognize the
> fact
> > that the C build system is not just a developer productivity tool, it is
> > also the installer for our end users. And before anyone says our end
> users
> > will just use yum or equivalents, all those packaging tools *also* depend
> > on our build system both directly, and because we can't even supply a
> > release for packagers to consume without a reasonable amount of direct
> > install testing. To a good extent a standard looking C source tarball is
> > pretty much the equivalent of a jar or jar + pom file in the Java world,
> > it's really the only platform independent means of distribution we have.
> >
> >
> It would be helpful if you could enumerate requirements which you believe
> to be missing and add the to the existing wiki page.  I don't think anyone
> is suggesting that the make install step should be broken in the source
> tarball, so it's a little unclear to me the problem you are trying to
> highlight above.

I believe it was suggested at one point that we not have a C source tarball
but just export the entire tree as a single source tarball. This strictly
speaking would not break the make install step, however it would have a
serious impact on our ability to leverage others to test the C impl. Anyone
downloading this would need to understand a great deal about the dual
nature of proton and how it is structured just in order to know that they
can ignore half the tree. Compare that with a standard C source tarball
where I can hand it off to someone who knows nothing about proton and
simply tell them to do a make install and then run one test script. Given
the latter structure to our release artifacts there are *significantly*
more resources we have access to in order to perform the testing necessary
to do a quality release.

I'll take a stab at distilling some requirements out of the above scenario
and sticking them onto the wiki page, but I actually think the scenario
itself is more important than the requirements. There's no disagreement
that it would be nice to have a very standard looking C source tarball with
minimal dependencies and so forth that can be used in the above manner,
it's simply the relative priority of the requirement when it conflicts with
developer convenience that is a source of contention.

> > It's also probably worth noting that perhaps the biggest issue with
> system
> > tests in Java is not so much imposing maven on proton-c developers, but
> the and nottest against Java code running elsewhere.
> > fact that Java may not be available on all the platforms that proton-c
> > needs to be tested on. My primary concern here would be iOS. I'm not an
> > expert, but my brief googling seems to suggest there would be significant
> > issues.
> >
> >
> So, I think we probably need to consider what sort of tests are required,
> and which languages it is appropriate to write any particular type of test
> in.  For me tests in Java have some advantages over Python tests. Firstly
> they allow interop tests between the two implementations within the same
> process

Can you elaborate on the benefits of this? It seems to me when it comes to
interop testing that, to the extent you can get away with it, over the wire
tests would be preferred. For example you could run proton-c on iOS via
Java tests running on another system, to say nothing of testing against non
proton implementations which would necessarily need to be over-the-wire.

> and secondly they will also be able to be used against any future
> pure JavaScript Proton implementation (something we have planned to do but
> not yet embarked upon).

This is also true of python tests. In fact the whole point of a python test
suite is that you can run python, java, and javascript all within the same
JVM using purely Java tooling.

 A third issue for me is that when we start to
> attempt more granular testing of things such as error handling, I will want
> to ensure that the user experience is identical between the pure Java and
> JNI binding implementations of the Java Proton API... if the tests are
> being run through a second translation into the Python API then this is not
> easily verifiable.
Why is this not verifiable? The proton API in both python and Java have
largely identical object models, and both languages support nearly
identical exception models. I would expect that testing for specific
exceptions from the Python API would quite directly force you to translate
those exceptions with high granularity from the Java API, and then you
could run the negative test suite against both pure Java and Java/JNI.

> As a final aside, on the standard development environment many of us have
> to work with, the installed version on Python is too old to support the
> current Python client (lack of UUID, etc).

It's true there are a few pieces we depend on that aren't available in
python 2.4, however the qpid python client has a compat layer that provides
these pieces if they are not there and makes the 2.4 vs new python
experience identical. Pulling this into proton would be quite trivial.

> Personally I think the more tests we have the better, and it's more
> important to encourage people to write tests than to force the use of a
> particular language to write them in.  I'd also suggest that we should be
> writing at least some tests for each of the idiomatic bindings.

I never said we shouldn't write tests, however as you say above it's
important to understand what kind of tests need to be written where.
Writing engine/protocol tests in every binding has very limited utility
since exercising the engine from one binding will run the exact same code
as exercising it from another, however writing tests for a binding that
verify that types are decoded properly is quite useful as those tests would
actually be exercising the binding code itself which is unique to the
binding language. As you point out, checking that exceptions are mapped
into a specific binding in a particular way is also binding specific in the
general case, however I think in the specific case of the Java binding a
test suite written in python could easily be run against both the Java and
JNI implementations and provide us with a high degree of confidence that
they behave identically. The only case where behaviour could differ is if
the python binding where to merge two error conditions into one, but this
would presumably just be a bug as the python binding itself would be losing
relevant information also.

In general I'm quite interested in what you think fundamentally can't be
tested from python vs what you think could be but is cumbersome or simply
blocked for reasons X, Y, and Z. I think a clear statement of this would
help identify areas we could improve (e.g. python 2.4 support) and also
help inform us what kind of Java tests should be encouraged.


Reply via email to