Hi,
Recently we've been building client distributions for our customers
but have run into some interop problems; in particular a lack of
interop tests to confirm that the clients really do work as intended
is impairing progress. We need a suite of interop tests that implement
the same test code accross all of the clients in Qpid and we need to
be able to run this regularly (in a fully automated way ideally). If
there are X clients in Qpid and Y brokers, then we need to perform X
times Y times X test suite runs to ensure that every type of client
can talk to every other type through each of the broker
implementations. This is going to be my focus in the immediate future
and, I'm sure you'll agree, an important thing to achieve in order to
demonstrate the whole of Qpid working together as a coherent product.
As we have different developers working in different implementation
languages I would like to propose an interop testing spec that defines
exactly how test clients should behave, so that we can write a suite
of tests that can talk to each other succesfully. There are some
existing tests, for example, topic_publisher/listener, in c++ and
java, but when I came to run these tests to interop test between c++
and java clients, I found that despite the similar names and closely
related code (perhaps at one time identical?), they could not in fact
talk to each other because they were using different topic names and
behaving in slightly different ways. Please read this spec and give
some feedback on it. With a view to reaching a consensus fairly
rapidly, I propose to incorporate the feedback into the draft spec to
arrive at a working spec over the next few days. I will then post the
working spec which we hopefully all agree on and are prepared to buy
into.
I'm particuarly interested in getting some constructive feedback on
this. If you disagree with something, please also suggest an
alternative way of doing it, that you feel will be better, more
reliable, easier to implement, or whatever. Thanks.
Rupert
=====================================
Qpid Interop Testing Spec. Draft 1.
Draft 1. Rupert Smith. Document started.
Introduction:
The requirements in this specification use a common format, an
example of which is given below:
RE-1. Sample Requirement. A brief descritpion of the requirement.
The requirements are numbered sequentially from 1.
Purpose:
* Test sending from and receiving by each of the clients in Qpid over
both of the broker implementations.
* Make tests robust enough to run as part of an automated build. The
scripts should pass or fail, not hang, wait forever, run out of memory
or otherwise cause an automated build process to be flaky.
* Be capable of running the full test suite on several machines in a
hands free way. In particular C++ tests need to run on unix and .Net
on windows.
* Run just a few tests to begin with. More can be added later, I'm
interested in getting the test framework established as quick as
possible. Just
minimal p2p and pub/sub tests to begin with.
Constraints:
IOP-1. Operating System: The test client scripts must run on Unix and
Windows. If a test client implementation is only available on one of
these platforms it only needs to run on its supported platform.
IOP-2. Scripting Language: Each test client must be startable from a
Unix shell script. Tests run on Windows will use Cygwin to run these
scripts. There is no need to support Windows .bat scripts.
Functional Requirements:
Introduction.
These requirements describe the behaviour of test clients for interop
testing between the different client implementations in Qpid. Each
client is expected to be a program that is capable of sending test
messages to other clients and receiving and responding to test
messages received from other test clients. The clients are not to be
run as seperate programs for the sending and receiving parts for the
sake of convenience in being able to run the clients as part of an
automated build. The clients will listen for control messages sent by
a master client on a topic, to tell them when to begin their tests and
when to shutdown. The clients will send control messages to each other
individually on a direct exchange in order to communicate about
individual test cases being run.
Common Requirements.
IOP-3. Directory Structure. All scripts to start and stop brokers and
run test clients will be placed in a directory structure underneath a
top-level directory called 'interop' that sits at the top level of the
Qpid project.
IOP-4. Test Output Format. Output in junit xml format (because a lot
of automated build software understands this format). There doesn't
seem to be a schema or DTD for this format but it is simple enough.
See Appendix B for an example. Each sending test client will output a
test suite report for each receiving test client that it runs the test
cases against.
IOP-5. Terminate On Timeout. Each client will keep a timeout count.
Every time it gets a message it will reset this count. If it does not
hear from the broker at all for 60 seconds then it will assume that
the broker has died and will terminate with test failures.
IOP-6. Default Virtual Host. All test clients will use the virtual
host '/test' for all tests.
IOP-7. Default Control Topic. All test clients will use a control
topic with the routing key 'iop.control' on the default virtual host
on the default topic exchange. This control topic is used for sending
the start messages (IOP-23), declare available messages (IOP-24), test
complete messages (IOP-25) and terminates messages (IOP-26) only.
Use Case 1. Starting a Broker.
Run the broker start script.
The script starts a broker running and tries to connect to it (or
otherwise ping it) until it is verified to be running.
Once the broker is verified to be running the script terminates with
no error code.
Failure path: The broker fails to start or does not appear to be
running after a timeout has passed. The script fails with an error
code.
IOP-8. Broker Start Script. The java and c++ brokers will define
scripts that can start the broker running on the local machine, and
these scripts will be located at interop/java/broker/start and
interop/c++/broker/start. The Java and C++ build processes will
generate these scripts (or copy pre-defined ones to the output
location) as part of their build processes.
IOP-9. Broker Start Timeout. If a broker fails to start within 60
seconds its start script will timeout. Script will terminate with
error code 1.
IOP-10. Broker Start Succesfull. When the broker starts succesfully
the script will terminate with error code 0.
Use Case 2. Stopping a Broker.
Run the broker stop script.
The script terminates the broker that was started with the start
script if it is still running.
Failure path: The broker won't terminate. The script fails with an error code.
IOP-11: Broker Stop Script. The java and c++ brokers will define
scripts that can stop the broker running on the local machine, and
these scripts will be located at interop/java/broker/stop and
interop/c++/broker/stop. The Java and C++ build processes will
generate these scripts (or copy pre-defined ones to the output
location) as part of their build processes.
IOP-12. Broker Stop Timeout. If a broker fails to terminate within 60
seconds its stop script will timeout. Script will terminate with error
code 1.
IOP-13. Broker Stop Succesfull. When the broker stops succesfully the
script will terminate with error code 0.
Use Case 3. Starting a Test Client.
Run the client start script. The caller will pass in address of the
broker to connect to and the number of messages to send per test.
The script starts a client running.
The client starts running but waits for further instruction before
running its tests.
The start script will terminate but leave the client running as a
forked process.
Failure path: The client will not start, or fails to connect to the
specified broker. The script will termiante with error code 1.
IOP-14. Client Start Scripts. For each client implementation,
<client>, there will be a start script located at
interop/<client>/client/start. The build processes for each client
will generate these scripts and output them to this location as part
of their build process.
IOP-15. Client Start Timeout. If the client fails to start and
connect to the specified broker within 60 seconds the script will
terminate with error code 1.
IOP-16. Client Start Succesfull. When the client starts successfully
its script will terminate with error code 0.
IOP-17. Client Start Broker Url. The -b <broker_url> option will be
used to call the start script to connect to the broker specified in
the url.
IOP-18. Client Start Messages Per Test. The -m <num_messages> option
will be used to tell the client how many messages to send per test.
IOP-19. Client Number of Receivers. For topic testing each client
will simulate the behaviour of many clients listening to the same
topic. The number of recievers per test client for topic tests will be
specified by the -r <num_receivers> command line option.
Use Case 5. Overall Test Procedure.
Start a broker running using its start script as described by Use Case 1.
Call the start all clients script on each of the machines where
there are clients that are to be tested. The caller will pass in
address of the broker to connect to and the number of messages to send
per test.
The start all script will scan for all start scripts located under
interop/<client>/client/start and call each of them with the broker
and number of
messages arguments. This performs Use Case 3 for each client.
Call the master test script. The caller will pass in the address of
the broker to connect to.
The master test script will send out a begin message on the control topic.
Each of the clients will receive the begin message.
Each client will post a message to the control topic declaring its
availability to test and locations on which it is expecting test
messages to be sent. This message will contain the routing key on the
default direct exchange to which the client recieves control signals,
the routing key on the default direct exchange to which it expects
test p2p messages to be sent, and the routing key on the default topic
exchange to which it expects test pub/sub messages to be sent.
Each client will hear on the control topic what other clients are
available and the addresses on which they expect to be sent messages.
Each client will run test cases 1 & 2 against each other client.
Each client will post a message to the control topic declaring that
it has completed its tests.
The master test script will listen for each client to send its
completion message.
Once all completion messages have been received the master test
script will send out a terminate message on the control topic.
Each client will receive the terminate message and will shutdown.
Terminate the broker using its stop script.
IOP-20. Start All Script. There will be a start all clients script,
located at interop/startall. The startall script finds all client
starts scripts under interop/<client>/client/start and calls them.
IOP-21. Start All Script Options Forwarding. The start all script
will take the same command line options as the client start scripts
and will pass these command line options on to them.
IOP-22. Master Test Script. There will be a master test script that
kicks off the testing process once all clients have been started. It
is to be located at interop/testall. It will start a master test
client that sends the start and terminate messages.
IOP-23. Start Message. The master test script will send a start
message to begin the test procedure. All test clients will listen for
this message on the control topic. The start message will be
identified by the header field, "CONTROL_TYPE", having the value,
"START".
IOP-24. Declare Available Message. Every test client that receives
the start message will respond by declaring its availability to run
interop tests with each other client. The client will send a declare
available message to the control topic. The declare available message
will be identified by the header field, "CONTROL_TYPE", having the
value, "DECLARE". The client will also declare the direct exchange and
topic exchange routing keys on which it expects to be sent test
messages and per client control messages. The client will also declare
a unique name by which it can be identified (see IOP-27). The declare
available message will contain the following header fields with this
information:
"CLIENT_NAME", "<client_name>" (see IOP-27 for rules about
the client name).
"CLIENT_DIRECT_CONTROL_KEY", "<client_name>.control" (see IOP-28)
"DIRECT_KEY", "<client_name>.direct" (see IOP-29)
"TOPIC_KEY", "<client_name>.topic" (see IOP-30)
IOP-25. Test Complete Message. Once a test client has run all of its
tests it will declare itself ready for termination. It will not
terminate immediately because it may still need to respond to test
messagesd from other clients. It will send a test complete message to
the control topic. This message will be identified by the header
field, "CONTROL_TYPE", having the value, "COMPLETE". The client will
also post its name in the "CLIENT_NAME" header field.
IOP-26. Terminate Message. The master test script will listen for
all test clients to complete their tests. Once it has received a test
complete message from each client that declared itself it will send a
terminate message to the control topic. The terminate message will be
identified by the header field, "CONTROL_TOPIC", having the value,
"TERMINATE". Upon receipt of this message the test clients will
terminate.
IOP-27. Client Name. Each test client will provide a unique name for
itself that reflects its implementation language and distinguishes it
from the other clients. Clients should append a timestamp or UUID onto
this name to cater for the case where the same client is used multiple
times in an interop test. For example, the same client might be run on
two different operating systems, in order to check that it works
correctly on both.
IOP-28. Client Control Direct Key. Each test client will listen for
test control messages directed specifically to it on the default
direct exchange. The routing key for these messages will consist of
the client name (see IOP-27) followed by ".control".
IOP-29. Client Default P2P Test Direct Key. Each test client will
listen for test messages on the default direct exchange. The routing
key for these messages will consist of the client name (see IOP-27)
followed by ".direct".
IOP-30. Client Default Pub/Sub Test Direct Key. Each test client
will listen for test messages on the default topic exchange. The
routing key for these messages will consist of the client name (see
IOP-27) followed by ".topic".
Common Requirements for Test Cases.
IOP-31. Client Message Receive. Whenever a test client recieves a
message from another test client it will increment the total count of
messages received from that client since the last reset. The message
will contain the name of the sending client in the header field
"CLIENT_NAME", and the count will be held against that name.
IOP-32. Client Reset Message. Whenever a test client wishes to begin
a new test procedure against another test client it will request that
its message count be reset. It will send a reset message to the test
clients individual control queue. The control message will be
identified by the header field, "CONTROL_TYPE", having the value,
"RESET", and will contain the name of the sending client in the header
field "CLIENT_NAME".
IOP-33. Client Status Request Message. When a test client has
completed sending test messages it may request the count of actual
messages receieved from the test client to which it sent the messages.
The status request message will be send to the test clients individual
control queue. This message will be identified by the header field,
"CONTROL_TYPE", having the value, "STATUS_REQUEST", and will contain
the name of the sending client in the header field "CLIENT_NAME". The
message will have its reply-to field set to the routing key of the
individual control queue of the sending client.
IOP-34. Client Status Message. Upon receipt of a status request
message, a test client will send a status message to the individual
control queue of the status requesting client, the routing key of
which can be found in the reply-to of the status request message. The
status message will be identified by the header field "CONTROL_TYPE",
having the value, "STATUS". The header field, "MESSAGE_COUNT" will
contain the count of messages received from the requesting client
since the last reset as an unsigned 32-bit integer. The client sending
the status message will also place its identifying name in the header
field "CLIENT_NAME".
Test Case 1. Basic P2P Test.
The sending client sends a reset message to the receiving clients
control queue.
The receiving client will reset its message count to zero.
The sending client will send the required number of test messages to
the receiving clients test direct routing key.
The sending client will send a status request message to the
receiving clients control queue.
The receiving client will send a status message to the control queue
of the sending client (given as the reply to location of the status
message for convenience). This status message will contain the
number of messages received since the last reset.
The sending client will compare the messages received to the
messages sent and pass or fail the test accordingly.
Test Case 2. Basic Pub/Sub Test.
The sending client sends a reset message to the receiving clients
control queue.
The receiving client will reset its message count to zero.
The sending client will send the required number of test messages to
the receiving clients test topic routing key.
The sending client will send a status request message to the
receiving clients control queue.
The receiving client will send a status message to the control queue
of the sending client (given as the reply to location of the status
message for convenience). This status message will contain the
number of messages received since the last reset. This number will be
the number
of messages sent multiplied by the number of topic receivers being
simulated by the receiving client.
The sending client will compare the messages received to the
messages sent and pass or fail the test accordingly.
Waiting Room:
Contains ideas for future additions to this spec.
How I anticipate this being run as part of a fully automated build:
Will try to get a free licence for Anthill Pro 3 as they offer free
licences for open source projects. Viewtier Parabuild is another
possibility. Anthill Pro runs a central build server that does all its
work through build agents that can run on many boxes. It also lets you
define build workflows. I imagine running a Unix agent to build the
c++, java and python stuff, and a Windows agent for the .net stuff.
Will define a workflow that starts a broker on the unix box, then
starts all clients built on the unix and windows boxes in parallel,
then runs the entire test procedure across all clients, then
terminates the broker on the unix box. The agents send back the test
results to the central server.
Full testing of field tables. Make sure that every possible data type
is tested and confirmed to encode and decode correctly between all
client implementations.
Testing more of the protocol. Add tests to more fully exercise the
complete AMQP protocol.
Appendix A, General Notes:
Brokers that need to be interop tested: C++ and Java
Clients that need to be interop tested: C++, Java, Java 1.4
retrotranslation, C++, .Net 2.0, .Net 1.1, (Mono?), Python, Ruby.
Appendix B, Example of XML Format for Test Ouput:
I don't think there is a DTD or schema for this but the XML output
from JUnit looks like the example below. This is a convenient choice
for the output format from these test results even if the code does
not actually use JUnit (or cppunit or nunit) iternally, because
automated build servers generally understand and are able to produce
test reports from it.
Example:
<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="0" skipped="0" tests="18" time="0.02" failures="0"
name="org.apache.qpid.framing.BasicContentHeaderPropertiesTest">
<properties>
<property value="Java(TM) 2 Runtime Environment, Standard Edition"
name="java.runtime.name"/>
... (there were lots of properties).
</properties>
<testcase time="0.02" name="testRejectedExecution"/>
... (there were lots of test cases).
</testsuite>
