Rupert Smith wrote:
I'm particuarly interested in getting some constructive feedback on
this. If you disagree with something, please also suggest an
alternative way of doing it, that you feel will be better, more
reliable, easier to implement, or whatever. Thanks.
I think the approach is great. One thought that occurred is that by
offloading more work to the controller/master we minimise the amount of
framework code we need to write for each test in each language. The
controller would only need to be written in one language.
I like the idea of having a single client executable for each language
also, that can run a named test under a particular role. That makes it
easy to start up a given number of instances of different client
implementations without having to remember too many script options etc.
The controller can handle the test to run and the role each client
should take on.
I'd suggest we use an unspecified virtual host. Assuming we will be
using a dedicated broker instance for the tests we won't need the
isolation virtual hosts provide and just using the default make setup
easier in my view.
Attached are some notes that offer an alternative system. I find it
simpler and yet quite flexible, but as is often the case that might just
be because it fits more with my preconceptions. Its not as tightly
specified as your document but if you like the direction we can work on
the details a bit more. If you don't like the direction I won't be
offended, so please feel free to ignore it!
==Actors==
There are two types of actor in the test:
1. controller
A single controller will operate every test. The controller framework
only needs to be written in one language. The steps a controller
follows for each test are:
creates and binds an exclusive queue to the nameless exchange with
key 'control'. consumes from this queue.
sends out a 'invite' for a named test to amq.topic with routing
key 'control'. any listening clients that are able to be part of
that test respond with an 'enlist' message.
checks a queue exists for each enlisted client and binds them to
one of two new fanout exchanges created for the senders and
receivers on this test
sends a 'role-assignment' message to each exchange
wait for 'ready' messages from all enlisted receivers
sends a 'start' message to the senders exchange
waits for 'sent-report's from each sender(*)
send an 'end-test' to all receivers
waits for 'received-report's from each receiver(*)
compares actual and expected reports and prints results
deletes senders and receivers exchanges
(*)if not all reports are received in a given time, the controller
will send an 'end-test' message to all outstanding participants
2. participant
Each client is capable of participating in a set of named tests in a
particular role. Each client has a unique id. The roles are for now
'sender' and 'receiver'.
In general the clients follow the following steps:
create a queue named after clients id, bind it to amq.topic
with routing key 'control' and consume from it
listen for 'invite's to tests they are able to participate in
when such an 'invite' is received, send an 'enlist' message to the
control queue
wait for a 'role assignment' then carry out the relevant role
when a 'terminate' message is received, shutdown
The steps for the different roles are:
2(a) sender
waits for 'start' message
sends messages to a defined 'route'
sends a 'sent-report' to the master
2(b) receiver
prepares to consume messages from a defined 'route'
sends a 'ready' message to control queue
records all received messages
detects completion by receiving an 'end-test' marker
on completion sends a 'received-report' to the master and carries
out any cleanup required
==Control Messages==
All control messages have the 'control-type' property set to the name
of the message type. E.g. a sent-report contains 'sent-report' in the
control-type property. Unless otherwise specified all control messages
are empty messages.
A sent-report is:
A text message with a line for each sent message using the tests
stringified format.
It also contains a string header indicating the senders id.
A received-report is:
A text message with a line for each received message using the
tests stringified format.
It also contains a string header indicating the receivers id.
A role-assignment message is:
A text message containing the test name and a string header 'role'
containing one of the valid roles for the test (i.e. sender or
receiver to begin with). It may also contain other properties for
paramterised tests.
An invite is:
A text message containing the name of the test that the master
wants to run. The 'control-type' property is set to 'invite'. The
parameters of the test may also be specified in the invite
allowing participants that don't support them to opt out.
An enlist message is:
A text message containing the name of the test.
It also contains string properties indicating the client identity
and a description of the client (language, platform or whatever
seems relevant).
==Tests==
A named test defines:
The details of the route through which messages will be sent, in
particular the exchange, routing key and headers to be used by
senders and the details receivers need to subscribe (e.g. queue to
consume from, whether to create the queue, whether to bind the
queue etc).
The details of the message content and properties that will be
sent.
A stringified format for those messages.
Aspects of a test may be parameterised, details are defined on a
case-by-case basis. Parameters are sent in the test invite and
also in the role-assignment. Both tests below take a message-count
parameter.
===Simple Pub/Sub===
A simple pub-sub test where messages are sent to amq.topic with a
controller specified routing key. Receivers create their own exclusive
queues and bind them to the amq.topic with the supplied topic name,
they consume from them in no_ack mode. On cleanup, receivers delete
their queues.
Messages are text messages containing the string 'Message %d' where %d
is replaced by a monotonically increasing number. They also have a
string property containing the senders id. The format in which they
are reported is a line with the client id followed by a '#' followed
by the message text.
The controller can decide to have a single publisher or multiple
publishers. At the end of the test it checks each sender sent the
correct number of messages and each receiver received the correct
number. It also checks the values of messages sent against those
received by a comparison of the lines in the sent- and received-
reports as well as their relative order.
The topic name is passed as part of the role-assignment to allow tests
operating with different topics e.g. to test wildcard matching or to
ensure two topics are kept distinct. It is passed as a string property
called 'topic-name'.
===Simple P2P===
A simple point-to-point test where messages are sent to the nameless
exchange with a routing key equal to the supplied queue name (passed
in on role-assignment). The controller creates the queue to use and
binds it to this routing key (it is also responsible for deleting it
when the test is over). Receivers create a consumer on that queue. On
completion receivers cancel their consumers.
Messages are sent and reported in the same format as for Simple
Pub/Sub.
The controller can start the senders before or after the receivers. It
can create varying numbers of each role. At the end it ensures that
every message that was sent was received once and checks for relative
ordering. It also checks that no messages remain in the queue. (It
could also report on more statistical checks such as reasonably fair
distribution).
This test has an ack-frequency parameter, passed as an int property on
the role-assignment. If 0 it implies that no_ack should be used by
receivers. Else it implies that an ack should be sent every x
messages, acking all messages to that point. An ack should be sent on
completion for any outstanding messages.
It also has a txn-frequency parameter, again passed as an int
property. If 0, no transactions are used. If non-zero a transaction
should be used by both senders and receivers and they should commit
after sending/receiving that number of messages. All transactions
should be commited on completion.
===Future work===
Extra tests can be defined that do more complicated things, perhaps
involving different properties, new roles, bi-directional
communication rollback etc etc. The two simple tests above should be
relatively easy to code in each language and should still allow a fair
bit of flexibility in how they are put together. The code for each
scenario only needs to be defined in the controller framework which we
only need to write once in one language.
