Tomas Restrepo wrote:
Hi Rupert,
I'm particuarly interested in getting some constructive feedback on
this. If you disagree with something, please also suggest an
alternative way of doing it, that you feel will be better, more
reliable, easier to implement, or whatever. Thanks.
I think this is a fantastic idea and very needed. Some thoughts:
1- Would this be mostly aimed at testing client-client interop,
client-broker interop or both? It seems to me much of the implementation
needs you specify seemed to be aimed at client-client, but maybe I'm
mistaken.
I think it is both. You raise a good point though, in that if the test
infrastructure assumes a degree of interoperability to begin with, the
tests it runs are only worthwhile if they test interoperability in a
more advanced sense. That was the assumption I was making; allowing easy
creation of tests that give us confidence that systems written using
different pieces of the product will all work as expected rather than
just verifying that clients can 'talk to' brokers in a basic way.
2- Personally, I'd favor an approach a bit more like Gordon's idea of it
being more "centrally controlled" by the controller. Start a client-test
process, launch the controller, and do everything from there. It would be
simpler to create, run and maintain, I think.
3- I think the initial client to broker connection tests might want to be
handled a bit differently (maybe a set of more automated, but regular,
integration/unit tests for each client testing different success and failure
connection conditions).
Yes, I agree. In c++ we have a really simple client test program that
connects, sends itself a message then stops. Something similar for each
language would give a simple test for the ability to connect, send and
receive in basic form. Thats probably the first step before any test
framework is run. Maybe we should also have a dummy test that doesn't do
any real work but just allows the operation of the framework to be verified.
My main reason for saying this is that I think it might be more awkward to
cram the connection-level tests into the kind of structure proposed, and
even more if we went with a more central architecture such as the one Gordon
proposed.
Yes, I completely agree.
More to the point, one of the basic ideas you propose is already using
broker communication and queues to connect the test controller with the
individual test clients, meaning already you assume a fairly high-level of
interoperability being possible between clients and brokers (and even
between clients seeing as how the controller would be written in a single
language and each client in a different one).
This is also one of the main reasons I ask whether the tests will mostly
target client-client scenarios or client-broker scenarios (as for most of
the infrastructure it seems to assume the latter already works pretty well).
Then again, maybe I'm just missing something :)
I think you are right. Any framework based on AMQP itself should
minimise the functionality it relies on. There needs to be some test of
basic connectivity separate from that to ensure that a given client can
be relied on to work within the framework. The value of the framework is
then dependent on what other useful aspects it can test. There is always
a danger of over-engineering.
However assuming that we want to test more than just single client to
broker interactions, there are some common aspects that you don't want
to have to keep re-writing, there are co-ordination issues such as the
ones Alan raised, verification and reporting of results etc. I also
think that there is useful interop testing to be done around features
not required by the framework e.g. acks, txns, more advanced routing
patterns etc.
