On Apr 7, 2009, at 11:04 PM, Toby Biddle wrote:
A new online, unmoderated user testing tool has recently launched -
www.loop11.com. Anyone used it have any thoughts?
We haven't used this tool in particular, but, from their site, it
looks similar to a slew of other tools on the market.
These tools are limited in value because of four key factors:
1) The pool of invited participants is critically important. In
Loop11, it seems you have to invite your own pool , which means you
have to use standard recruitment techniques to source, schedule, and
incent participants in the study. This will probably triple (or more)
the costs. (Many unmoderated tools offer their own pre-recruited
pools, which keeps costs down, but are often low quality participants,
such as people who only participate to get the incentive and don't
really use the design.)
2) You are limited in the tasks your participants can perform. For the
software to work, the site has to know when a task is completed. For
example, when evaluating a travel site, you have to know what page the
user will end up on. If the confirmation page for a trip booking is
computer generated, this might not be possible. Even if it is, can the
system tell if all the values were properly entered?
3) We know from our research at UIE that participants who are actually
interested in the task (for example, currently planning a vacation in
Paris) will behave substantially differently than those who are asked
to pretend to do a task. They take more time, are more discriminating
on the results, are more likely to be frustrated when key information
is missing, and are more likely to be delighted when the design meets
their needs. Yet, these systems usually require that every user take
the same path through the system, which means recruiting people with
identical interests (every participant has to be actively planning
their vacation to Paris and desiring the same dates & hotel
requirements).
4) The site reports standard analytic measures: time on task, "fail
pages", common navigation paths. But it's extremely difficult to come
to the correct inference based on these measures. For example, does
longer time-on-task or time-on-page imply frustration or interest?
Does a deviation from the common navigation path imply clicking on the
wrong element or curious exploration of additional features? Without
talking to the individual, it's hard to even know if a reported
measure is good or bad, let alone the action the team should take
based on the reported result.
In the ten years since I first started seeing these tools on the
market. I've never seen results from a study that the team could
actually interpret and act on. In one study a few years back with a
major electronics retailer, we conducted an in-lab study with 18
highly-qualified participants that was comparable to a 60-participant
Netraker (a Loop11 competitor from the past). The task was to find the
laptop computer of your dreams and put it in the cart.
In our study, all 18 participants were in the market to buy laptops,
had spent at least a week thinking about the laptop they wanted and
its requirements, and were given the cash to make the purchase (they
would keep the laptop after the study). In the Netraker study, they 60
randomly selected participants from a panel of thousands who
reportedly were in the demographic groups of the site (unverfiable)
and hadn't thought about laptop purchases until the instructions for
the test had popped up.
In the Netraker results, 94% of the participants completed the tasks
and the average time was 1m 18s. In our study, only 33% of the
participants completed the task and the average time was 18 minutes.
Why do you think there were such striking differences? Which study
would you pay more attention to?
Beware of VooDoo measurement techniques.
Hope that helps,
Jared
Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: [email protected] p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool
UIE Web App Summit, 4/19-4/22: http://webappsummit.com
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... [email protected]
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help
