An interesting summary of the research Kasper has done into automated
testing tools.
Michelle
Begin forwarded message:
From: Kasper Galschiot Markus <[email protected]>
Date: April 7, 2010 7:39:53 AM GMT-04:00
To: CollectionSpace Work list <[email protected]>,
[email protected]
Subject: [Work] Automated testing -- Tools discussion
I have been browsing around in the quite extensive amount of free
web test automation tools around. Everyone seems to have their own
preferences, and there are pros and cons for each tool. Below I'll
try summarizing my observations (not experience, but what I have
read). In the very bottom, I posted my personal opinion of choice.
I'm sending this mail to the work list, because any experience,
rumors, comments, suggestions, etc. would be highly appreciated.
Also comments on the importance of: speed of tests, reality of
tests, ease of programming, etc, as well as suggestions for other
tools I should look into would be appreciated.
----
Selenium: Probably the best known tool.
Pros:
It automates/scripts real browsers, which is an advantage as you get
close to reality.
Well established and documented
GUI record tool for firefox, making it easy to write test (for all
browsers)
Support multiple programming languages (for writing the tests)
Has been praised for being the easiest to use/get into
Supports ant
Has grid functionality to distribute the tests (to a certain extend)
Cons:
Events are simulated, which is sometimes buggy (based on rather old
articles, and justins (fluid) experiences X months ago, might have
changed).
Needs actual running windows+mac machines to run against
Slow compared to emulated browser approach (can be somewhat remedied
by running tests in parallel)
Disregards JS errors (unless they affect the asserts)!!!!
Potential problem with waiting for pages to load (think this has
changed)
Canoo Webtest: Also widely known tool.
Pros:
Do not need specific OS, as browsers are emulated (uses HtmlUnit)
Record tool for writing easy tests
Well established/documented
Supports ant
Easy to integrate/setup continuous testing
Very good reports
Fast
Cons:
Does not use real browsers, but emulates them using HtmlUnit. This
means some errors might potentially be not be caught.
A bit steeper learning curve than Selenium
Javascript support not as good as in "normal" browser
Doesn't accept (too) badly formed html (Dont think this will be a
problem)
Doh.Robot: A smaller project, brought to my attention by Justin.
Pros:
Does not simulate mouse/keys, but makes actual system calls using
java's robot
As close as can be to real users (except for embedded scripts)
Cons:
Not too well documented/established
Unsure if updated or discontinued
requires code to be inserted on the page
Sahi:
Pros:
GUI record for writing tests
Cons:
Small community
Underdeveloped
Confusing Interface
Watir:
Pros:
Multi browser (& OS) support
Has a rich API
Has a ‘Simple’ class (for non-tech users)
Watij & Watin (Java & .NET)
Cons:
Have to learn Ruby (unless you choose Watij or Watin)
Every browser requires a different library
Selenium 2:
Pros:
Has both standard selenium and HtmlUnit available for testing
Circumvent browser restrictions by using whichever mechanism is most
appropriate to control the browser. For Firefox, this means that
WebDriver is implemented as an extension. For IE, WebDriver makes
use of IE's Automation controls, etc.
WebDriver can make use of facilities offered by the OS (allows to
get closer to a real user, similar to robot i guess?!)
Cons:
Alpha state
Sparse documentation, and probably somewhat buggy
Steeper learning curve due to amount of features and lack of docs
---
From what I can tell, Selenium and Canoo Webtest seems to be the
most appealing choices, despite their shortcommings. It seems that
Webtest receives less criticism than Selenium, but it is hard to
tell whether this is due to Selenium being used/compared to more
often. Generally, both people using Selenium and Webtest are
satisfied. The others; Doh Robot, Sahi and Watir all have their
advantages (except for Sahi :( ), but for various reasons didn't
appeal to me (too little documentation, steep learning curve,
lousier version of Webtest/Selenium). There is also the option of
Selenium 2, but the problem with this is that it is still in alpha,
and the documentation is rather sparse. Also, it might be kind of
buggy.
In many ways I am split between Selenium and Webtest, but I guess I
am leaning towards Selenium. There are two main reasons for this:
Using real browsers is a lot closer to reality, which I weigh high
(some of) the shortcommings of Selenium will be fixed in Selenium 2.
I take it that our tests written for Selenium will be compatible
with Selenium 2. When S2 will be at a decent stage it is easy to
make the switch.
The major problem with Selenium is the need of access to windows/mac
machines (windows is doable, using Vms, I hate mac :).
Another major problem is the speed. This can hopefully be remedied
using Selenium Grid.
As a final note, I guess I should mention that I might be a bit
biased towards Selenium, since I knew that first, have read more of
the documentation, etc.
_______________________________________________
Work mailing list
[email protected]
http://lists.collectionspace.org/mailman/listinfo/work_lists.collectionspace.org
------------------------------------------------------
Michelle D'Souza
Software Developer, Fluid Project
Adaptive Technology Resource Centre
_______________________________________________________
fluid-work mailing list - [email protected]
To unsubscribe, change settings or access archives,
see http://fluidproject.org/mailman/listinfo/fluid-work
