Last week Ian Wells send a message to the [updatesteam] list about a
problem we recently
had with some updates to SME8 having slipped through with insufficiently
tested and
users suddenly finding that something did not work any more after the
update.
In my experience in software development, comprehensive or fully testing
a new
release is a huge effort and quickly becomes very repetitive.
Unfortunately that
is something we humans are not very good at and hence testing tends to
be limited
to spot test of areas we expect to have changes or affected and even
there we
may overlook things.
Test automation will go a huge way towards ensuring SME is solid and has
received
a high degree of validation before release.
As most of functionality of SME (SSH, FTP, HTTP, IMAP, SMTP, POP, SAMBA,
etc.) is through
a network connection or a web page (server-manager) it should be
possible to test most of
that functionality from an external test host with some sort of script
or script
controlled browser (e.g. with Selenium).
Then with VirtualBox having a great great command line based API it is
possible to
create/configure a new VM and start/stop it all from a script. That API
also provides for
sending keystrokes and capturing the console display (as a picture, but
with OCR we can also
capture the text of it). VirtualBox also has a range of options for
different network
configurations and this makes it possible to also test both server only
and server-gateway
configurations.
Then I came across robotframework
(http://code.google.com/p/robotframework/).
It's a generic test automation tool for acceptance testing, under active
development and
to me looks like an almost ideal tool for use with SME automated testing
(although I don't
have any personal experience with it, but like to try).
I think that if we start to build a test suite against the stable SME8
release, this
then also gives us a test suite for SME9 as well.
So this post is really to see what the rest of our SME community thinks
about all this
before I go off on a tangent and if there is anybody that likes to join
me in developing
some of these automated test.
Also, I like to hear what people think about what the most critical SME
configurations
and tests would be for a SME release smoke/validation test so test
development can be
prioritised.
Kind regards,
Marco
--
Marco Hess
Through IP Pty. Ltd.
www.through-ip.com | [email protected]
p: +61 407 78 55 66 | f: +61 8 8121 6191
_______________________________________________
Discussion about project organisation and overall direction
To unsubscribe, e-mail [email protected]
Searchable archive at http://lists.contribs.org/mailman/public/discussion/
